What can firmware engineers learn from software engineers?

Question

Judging from my knowledge of the history of firmware engineering tools, practices etc. It has consistently lagged behind the software engineering field by several years. For example, as far as I can tell there is still a fair amount of debate in the firmware world as to whether C++ is actually worth using for our applications, and some C++ compilers are noticeably absent (microchip?!?). I imagine that in large part this is due to the differences in requirements between firmware and software. Again, judging from history, it seems its only a matter of time before the properly vetted tools and techniques make it into the firmware world.

What methods, tools, best practices etc that modern software engineers use regularly, could firmware engineers also leverage to improve their craft?

Specifically I'm thinking along the following axes (but don't let them limit you):

• Improving code cleanliness/maintainability
• Reducing defect introduction and improving detection
• Improving documentation
• Requirements management
• Improving reusability

I'd also love to see embedded shops answer or comment on the answers to provide feedback about theoretical feasibility or, better yet, personal experiences.

UPDATE
I'm especially interested in jumping ahead of the curve a little bit. So relatively new stuff that has been vetted reasonably well (works well for most people), like C++, TDD, etc. What do you use all the time and love?

UPDATE 2
I'm getting a lot of good general programming advice in the answers so far, which is great, but I'm really looking for more unconventional approaches that have proved successful for people. I'm trying to tease out the Agile practitioners, the TDDers, and the rest of you who have tried stuff and seen it pay off in spades or fail horribly. As a software engineer has there been a tool or practice that you've adopted in the past several years that has had a remarkably positive or negative impact?

Answer 1

• Source control
• Unit testing (TDD)
• Continuous integration (or nightly builds)
• Bug tracking

Firmware engineers I've worked with don't do any of these.

Unit testing may not apply very will to all kinds of firmware. I image its harder to unit test something when it's running on physical hardware. Depends if you have emulators available I suppose.

Answer 2

Not sure what degree of software is considered firmware (bios, driver, or utilities), but the standard complaint I hear is that the UI is non-standard. Realizing that UI matters and that it should follow some standards or tradition would be good thing.

As far as C++ goes, it's understandable to hesitate to get into it because there's a lot of overhead compared to C. C is almost like an assembly language with libc, whereas in C++ even a simple function call can be virtual. Implementing the full C++ runtime cannot be that easy given the complexity of the language.

There are too many things to list in terms of software development "best practices" (I hate that words). Why not start with The Joel Test: 12 Steps to Better Code.

The Joel Test

1. Do you use source control?
2. Can you make a build in one step?
3. Do you make daily builds?
4. Do you have a bug database?
5. Do you fix bugs before writing new code?
6. Do you have an up-to-date schedule?
7. Do you have a spec?
8. Do programmers have quiet working conditions?
9. Do you use the best tools money can buy?
10. Do you have testers?
11. Do new candidates write code during their interview?
12. Do you do hallway usability testing?

Answer 3

I have worked both as an embedded software engineer and as software developer. Being there in both worlds, I have learned that no matter how little resources your system has and what language you are programming, there are many things that can make your life easier.

The first thing is the tools you are using. In embedded software you only deal with the compiler/linker most of the time. There are more than these. Diff tools, regular expressions a scripting language, documentation tools save you a lot of time.

Another thing is code quality. One needs to follow style conventions, go through regular refactoring cycles and in general have in mind that code reading is done more often than code writing and it really pays to have more readable code.

Some times in embedded software we miss the design phase altogether. Embedded projects are usually not so big as desktop/server ones, but that is not excuse for not doing a proper design.

Software needs to be tested on its own and not only as part of the device. It really saves a lot of time to build a software simulator of your system, just for testing that the software meets the required specifications. It is much more expensive to do so when the whole thing, hardware and software is ready.

Answer 4

Firmware Engineering is quite broad. From PIC to DSP they all have varying degree of physical resources. These days DSPs are quite powerful (comparable with the CPUs ~5years old), can support large amount of memory etc. But then again you have PICS which operate with several kilobytes. More meagre the resources, the programmer has to use ingenious hacks to get the most out of the device. And the focus is on 'get it to work' rather than 'write elegant code'.

What I would like to see is good project management tools that incorporate code as well as documents as you have to refer to lots of datasheets, design docs which are scattered around the network and on email blobs etc..

I would also like to see better dev tools for DSPs (TI: please hand over CCS to some one good at making IDEs) , and more library makers using C++ (I'm looking at you ATEME) to build better libraries.

And also hardware engineers to have a better appreciation of Object Orientation rather than blurt out, 'if it's C++ it's going to be slow'.

Answer 5

Let me first address an assumption in your question. The assumption is that embedded software engineers (ESE) do not know or are not aware of modern software engineering practices and need to learn new practices. This assumption should be tossed out right away. I believe you would find the same statistical distribution of ESEs who keep their skills and techniques up-to-date as non-embedded SEs.

So, maybe your question becomes an array of questions like:

1. Why do ESEs use a separate code editor and a command line compiler and not an IDE?
2. Why is C preferred over C++ in the majority of embedded projects?
3. Why isn't there as much experimentation in programming practices in the embedded world?

The following paragraphs answer these questions.

1. ESEs usually use a specific code editor because it is a personal preference or it is what is used by his/her company. IDEs are not as common because ESEs work very closely to the silicon and not all chip manufacturers develop an IDE for their line of chips. Only the chips that achieve highest market penetration, such as ARM, have enough momentum to warrant the development of IDE-based tools. Furthermore, an IDE doesn't provide as much of an assistance to the ESE as it does to, say, a desktop developer. IDEs provide help with making the glue to the GUI or code-completion for large APIs; neither of which is commonly found or as standard as in the embedded world.

2. I'm sure there are better write-ups as to why C is preferred over C++ in embedded systems than I can make up on the spot. The short of it is that you use what works. C works and is more common (more programmers know C than C++). The other reason might be that the OO methodology was devised to help programmers solve large problems by abstracting the solution into manageable conceptual objects. In the embedded world, the problems are usually whittled down to as small a problem (and solution) as possible so that the embedded system becomes more reliable by having a smaller code base.

3. There is less experimentation by ESEs because an embedded product, in general, must be far less prone to error and have higher reliability than a desktop program. This means a rigid application of well-proven practices and more time spent in testing. Why? Because often there is no feasible path to upgrade the firmware of an embedded device; it's either impossible due to the system being deployed beyond reach or implausible because of the cost of updating millions of devices.

In conclusion, ESEs use tools and practices that best suit their needs just as non-embedded SEs do. As a practicing ESE, I believe the embedded software discipline is far more different than my non-ESE friends believe it is. So the apparent disparity of programming practices is not a matter of ESEs needing to learn modern practices, but non-ESEs needing to understand how different embedded programming is.

Answer 6

Automated testing
Do not visually scan your simulation outputs to check is everything is OK. You need comprehensive automated test because you are always going to miss something in that mass of waveforms.

Version Control
You are not going to remember what was the working version. Use Version control software so you know what to program that board with.

Bug Tracking
Your going to forget sooner or later. A bug log should contain the version(see Version Control) in which the problem was first detected and the version in which it was fixed.

Whoops I thought you meant firmware as in FPGA but the same is true for embedded software. If you already have these processes in place great else forget about unconventional approaches until get the basics right.

Answer 7

This is probably a bit out of context.
A short reference to a firmware column at Embedded,

• Bug-killing standards for firmware coding
• Some pages forward into the article, Ten Bug-Killing Rules

I have always found good articles on firmware engineering at Embedded.
Which probably many interested in this question have too...

Answer 8

What can firmware engineers learn from software engineers? Plenty!

I am surprised at how similar firmware development is practiced today as it was 25 years ago when we first started using C for embedded development. C was a big step forward from assembler, but there are many more lessons learned that firmware engineers can and should learn from. Yeah, some of the tools are better, but many practices are stuck in the 70s and 80s.

Embedded software development does add some additional challenges on top of the challenges faced by non-embedded developers. But all the principles and practices that skilled software developers use are applicable to embedded development. BTW: It's not just the embedded software developers that are not up on these state of the art practices, but many non-embedded software developers as well.

The people I know and have met doing firmware are by and large a very skilled group, working to solve difficult problems. Unfortunately, for whatever reason, many have not kept up with developments in the software world. I think it has to do with an imaginary barrier erected by firmware engineers.

Embedded and non-embedded developers speak different languages, but solve similar problems. Keeping embedded code independent from a hardware device is essentially the same as keeping application code independent of the UI or database. The underlying principles are the same.

Here are a few things that I think embedded developers should pay more attention to. Some of these principles and practices can be used right out of the box, while others might need a little adjustment to deal with the embedded challenges. If you want to substitute the word firmware for software below, go ahead, I don't really distinguish between the two.

Dependency Management

Dependencies between modules must be managed. Dependency from software to hardware is a special case that must be actively managed by the embedded software developer. If you don't manage the dependency, it will manage you.

In practice this means that only a limited subset of the software modules should have knowledge of the underlying hardware (and operating system). As the hardware evolves, and it always does, the investment in the hardware independent code can be preserved. See my ah ha! moment.

Robert Martin has written extensively on the SOLID design principles. Embedded developers should get to know them and apply them to their designs.

• S-Singled Responsibility Principle
• O-Open Closed Principles
• L-Liskov Substitution Principle
• I-Interface Segregation Principle
• D-Dependency Inversion Principle

These principles lead to designs that better stand the test of time. The SOLID principles encourage creating cohesive and independent modules. They are build on object oriented ideas, but can be applied to C. We have to stop the function-call data-structure free-for-all that is all too common in embedded C code.

C++ and OO languages

Why can't you use C++ and OO? Because they are too slow, or too big. What are the facts? C++ is a big, and mysterious language, but you don't have to use all of it. Take a look at Why are you still using C?

C++ makes up for some of the problems that C does not help much with like:

• Encapsulation and information hiding
• Programming to interfaces
• Substitutable objects
• Ad-hoc initialization

C++ can be used effectively for embedded development. Well you do need a C++ compiler, and the headroom. Maybe that is not possible in your world, or maybe it is the cost of doing business. Start by learning:

• classes - these are structs with member functions as well as member data.
• constructors - these make it possible to get initialization right, all the time.
• destructors - if you learn constructors, you must also learn destructors to keep the universe in balance.
• inheritance - use this mainly for defining interfaces that contain only pure virtual functions. Interfaces provide important dependency breaks and flexibility points. These are usually unjustly discouraged in embedded. There should be no mystery or prejudice here; virtual functions are function pointers under the hood. The alternative to effective use of interfaces is complex conditional logic, something that embedded C programs usually have too much of.

If embedded developers used those parts of C++ they could build more flexible design and not incur a high cost.

Test Driven Development

This might be the biggest game changer. I am glad to see other posts mention it too. TDD can help prevent defects now and in the future. To see why TDD might help take a look at The Physics of TDD.

Embedded does present some unique challenges for TDD. For example, TDD requires an extremely fast incremental edit/compile/link/run cycle. For many embedded developers this means careful Dependency Management and running unit test first on the target. See more about adapting TDD for Embedded.

With TDD, you are creating code that is thoroughly tested. The comprehensive automated test coverage acts as a safety net, allowing code to be changed safely as requirements change. Unintended consequences are immediately discovered.

Also, having the tests that you get almost for free, allow you to fearlessly refactor your code...

Continuous Refactoring

Code is written once, but read many times. Then it is changed and tweaked, leading to designs that degrade over time. If developers do not continually refactor to keep the code clean, it turns into a mess. Maybe some of you are dealing with that mess. TDD's automated tests enable low-cost and low-risk refactoring.

Continuous Integration

Automate your build process. Have it run with every workspace checkin. This is a challenge with the heterogenous tools sets often needed to get the compiled code into the target, but it is still the right goal.

The sooner the embedded developer knows that a change is somehow incompatible with some other work, the faster it can be repaired and the less time will be spend in painful merges.

Incremental Delivery

Find ways to split the work so that large painful integrations can be avoided, and design ideas can be tried early. Avoid splitting along architectural lines, focus on delivering slices of visible functionality.

Collaboration

Embedded developers! get out of there cubes and work together. How can you get better when you only see your own code? How can you improve when you are the expert on technology XXX, have mastered it and don't get an opportunity to work in different areas.

There is lots to learn out there. Are you responsible for being all you can