Using DevOps to Address Embedded Software Challenges

A perfect storm is brewing for embedded device software development. Embedded devices are proliferating, particularly as new connectivity options, such as 5G, make innovative connected device applications possible, and their use is expanding, taking advantage of new analytics and artificial intelligence capabilities. At the same time, the device lifecycle needs, such as responding to evolving cyberattacks, make it necessary to quickly create and deploy software changes.

These conditions place new demands on development and operations staff. Increasingly, DevOps is considered the best way to meet the market demands for more applications and new features, all made available in shorter times. RTInsights recently sat down with Graham Morphew, Senior Director of Product Management at Wind River. We discussed the challenges of embedded device software development, the differences between DevOps for traditional environments and embedded devices, and more.

Here is a summary of our conversation.

RTInsights: How is DevOps for embedded devices different from traditional DevOps approaches to developing software?

Morphew: When you look at traditional DevOps in an IT enterprise-based environment, you have a ubiquitous execution environment. You’re either running on Intel servers in the cloud, or you have something running on an Intel PC. Embedded is very different. Your end execution environment is usually a different architecture than the one you are using for development. There are more challenges because of the diversity of the end hardware and how you deploy to them. For example, when you’re building software for a cloud-based environment, you know it’s going in a Google Cloud, Azure, or AWS environment. When you’re building embedded software, the diversity of choice for deployment environment is almost infinite, and you also could have devices in remote locations.

So, there are many differences in terms of how you think about the software and how these differences translate into DevOps implementation challenges.

You must contend with specialized hardware, cross-compiling, cross-debugging, memory footprints, and security issues. You don’t have the almost limitless resources at your fingertips as you would in a cloud environment. There’s an island of execution that you need to keep in mind as you design these systems. You also have security to worry about. You don’t necessarily have physical control of the security of the devices. You may have to contend with physical tampering of the device.

Another challenge is that the data from these devices is more difficult to gather. In a DevOps environment, you want a continuous feedback loop. That’s easy if development is on servers that are under your control. When you’re talking about devices, they will likely be remotely distributed, and they may not be online all the time. So, many different issues come into play when comparing your traditional DevOps and DevOps for embedded software.

RTInsights: Why will the embedded device community move towards DevOps? Why is it becoming compelling, and what’s happening in the markets to drive this?

Morphew: With embedded software, there is a growing need for more frequent updates and higher quality. To get there, you need automation to help you along the way. Automation will play a big role in the future of DevOps and embedded software. If you go back a decade or two, there was a lot of focus on the hardware driving the capabilities of the device. Now, much, much more of the differentiating technology among device vendors is software.

Another factor pushing companies towards DevOps for embedded systems is something that we see at Wind River quite a bit: the changing demographics of software development.

There are two very distinct camps. You have your traditional embedded software developers and new graduates or developers entering the domain of intelligent devices from other software domains. The traditional developers tend to be older, and many are retiring. Like me, there are people who came into the market out of college at a time where you would look at hardware manuals, program registers, and things like that. That’s just not something that college programs spend a lot of time on now.

I have a son who’s college-age now. He’s programming in Python. He just learned C for the first time, and that was a big eye-opener. Python offers much higher levels of abstraction.

DevOps can help overcome this obstruction between the application environment and wanting to make that look like a familiar environment to developers of device software. The need to do this is that companies building devices have trouble acquiring new software development talent.

I was talking to someone we had hired as an intern recently, and he told us that many of his classmates want to go to Silicon Valley and work for companies like Facebook, Google, Apple, and Tesla. For companies in the industrial or aerospace and defense segments, it may be more of a challenge to attract the needed software talent that would come in and program embedded devices in a rudimentary C environment.

To overcome this, some companies think that giving that new generation of software developers an environment they’re familiar with will help. And that’s one of the reasons why Wind River is adopting a Visual Studio Code environment. Visual Studio Code is an environment that has seen rapid growth in popularity since coming into the market. Everyone that we talked to, coming from a new grad perspective, is very familiar with VS Code, and we see less experience from the new developer with older environments like Eclipse. So, sometimes you have to be where your audience is already.

RTInsights: What problems do companies have when they try to adopt DevOps solutions? And what are the key challenges for the embedded device realm versus other domains?

Morphew: The biggest challenge is the cultural shift that has to happen within companies. And this isn’t necessarily an embedded-specific challenge. It’s more deeply ingrained in some software development practices.

You have small teams, and in many cases, you have individuals who do very specific tasks. You need a level of collaboration and cooperation with DevOps that sometimes takes people out of the realm that they’ve become comfortable with for a number of years. You have to say, “Everyone’s working together.”

The Ops part of DevOps for embedded is a challenge because, in a traditional DevOps environment for the cloud, the Ops is pretty standard. You’re running a website or developing an application that does something through a cloud-based interface. When you’re talking about embedded, you are talking about a device in the field, and what that device does is specific to your company. In many cases, the device manufacturers are not the companies that are operating the devices. An equipment manufacturer may be selling its device to a big electrical company or a big manufacturer. Those companies are the ones operating the devices. Sometimes there is assistance from the device manufacturer, but it’s not a completely closed-loop the way you might see with a cloud-based solution.

There are toolset compatibility issues. Having a common development environment sometimes meets resistance. So that comes back to some of the cultural and management backing you need to implement these systems.

And then there is the hardware issue again. It’s a common theme in the embedded market. How do you get enough hardware to build out the automation environments needed to make DevOps a reality? That’s an ongoing challenge. Typically, successful customers have a mix of hardware and simulation to scale up their test processes.

RTInsights: Are there tools that would make the transition to DevOps easier?

Morphew: One thing helping companies get through this revolution is the availability of tools. Many of these tools are open source or have free versions. What you would often see was a consolidation around some flavor of Source Code Management, often a flavor of Git. Now, organizations have gone from having a very Source Code Management-centric solution to including more and more DevOps types of tools within their solutions. That helps companies make the transition.

There’s a lot of choice. You might argue sometimes there’s too much choice. The challenge that we see customers having now is, yes, there are many tools. How do I bring them together into a solution that works for me?

Today, many companies are starting projects where they’re forming teams internally to manage their transition to a more digitally-focused development environment. We see a transition happening now in the embedded space like what we saw with other technological transitions. In many cases, it’s very much a do-it-yourself mentality of, “Let’s build the perfect system for us, let’s customize it to our needs.” Such efforts are draining more and more resources out of the company just to maintain these things going forward. That’s not necessarily where they want to invest. That approach will likely evolve to one where people are looking to have another company maintain more of their environment over time.

Another area where, ultimately, companies can make their lives easier is to have clear separations between application development and maintaining the software platforms that they’re running on. It used to be you had a small team that did both things, and the application and platform merged into each other. But now, you need that clear separation to modularize your software and have people working on what they have the best skills for.

RTInsights: What are the industry drivers for solutions that make it easier to develop and test software for embedded devices?

Morphew: There is the convergence of the IT and OT worlds. You have devices connected to the Internet. This has been a big driver for companies to re-examine how they deliver software. Also, there are several industries where you have compliance-related requirements to update the software in devices. You see that in the medical field, where now you must prove that you can update your device if a security vulnerability becomes known. This can be a life-or-death scenario. You need to be able to prove that you can address such a problem if one comes up.

Such drivers are pushing companies to look at the processes they’re using regarding their ability to do remote updates. What we see is a lot of larger companies feel the threat from these digitally-native new and up-and-coming companies. There are even terms they use to describe it. You hear the term Teslafication. They’re saying they need to be more like Tesla and be a very, very software-focused business, as opposed to brick and mortar, steel, and iron type of thinking that is more associated with hardware. More and more, they must differentiate their product on the software that runs on the things they’re building.

The pandemic has also accelerated this trend. You have most of the software-focused workforce working from home. And in many cases, a significant number of employees are not going to return to the office after this is over. That’s a big shift. So, you need to change the way you think about working to make that situation productive for developers. That’s a challenge. It screams out, to some extent, for more collaboration tools and more standardization in terms of how things get done because you don’t have many people working face-to-face and collaborating in the more traditional way.

RTInsights: Moving on to a different issue, why have fast software iterations become a critical, competitive advantage in every industry? And how does that relate to that need for automated testing?

Morphew: I’ve gone through several projects that transitioned from a waterfall model to a more agile model. That ability to do continuous testing, automated testing is often the limiting factor in terms of increasing your productivity. If you’re going to run fast and you still want to maintain your quality, it’s a necessity. This is a particular area where having a digital twin of your end device lets you do a lot of the testing on it, and you can do it at scale.

One of the big advents that we see in terms of DevOps being applicable to embedded is the ability to take a simulation of the embedded device and then use it at scale in a cloud-based environment. In that way, you can have a hundred tests running simultaneously. You’re only limited by cloud resources. That’s one of the transformations we see a number of companies going through and something that they ultimately value quite highly.

We’ve had a simulation business at Wind River for many years. Some of the early adopters were doing a lot of this on their servers, scaling up large numbers of simulations. But when you move it to the cloud, you don’t have to buy new servers every six months.

You have control over the type of hardware you’re using and the amount of it. You can dial it up and dial it down depending on what you need at any one time, rather than having the capital expense of large hardware and IT teams to maintain it. Right now, we see a balance or sort of hybrid cloud environment, where some of the testing is done locally on-premises, and some of it is in a public cloud.

RTInsights: Are there any other points you want to bring up that we left out?

Morphew: When you talk about DevOps and moving embedded software development to a more cloud-native, cloud-focused environment, there are several things I’ve seen personally as a product manager that are big changes.

One of them is collaboration. I was trying to complete a Linux build. I’m not a tried-and-true Linux engineer. I was having some problems getting a build to work correctly. And while I was doing this, one of our Linux software architects could see that I had had several failed builds. And he contacted me over an instant messaging app and said, “Hey, I’ve seen that you’re having some trouble, I had a look, and you just need to switch the setting, and you’ll be good.  And I just went ahead and fixed it for you.”

If I were developing in a different environment where I had software installed on my PC, no one would ever know I was having problems. I’d have to go out and ask. Also, I most probably wouldn’t be able to recreate that same scenario. Most likely, I would have been stuck on it all day. And, I may have just given up after a while. So, the ability to get quick access to software that you don’t have to install locally, and being able to play in a common sandbox, to me, was a big eye-opener on how this changes how you work, just from essentially having these kinds of shared assets amongst your team.

RTInsights:  Anything else?

Morphew: The one thing we look forward into the not-too-distant future is to have digital feedback loops where you’re taking data off the device, taking data out of your development environment, and feeding that back to improve your software. AI and machine learning play into this as well. What kind of information can I get off these devices? How can I analyze it potentially with a large cloud-scale, big data type of a model or engine, and then feed that back into future software development? That could help me optimize a system as a whole.