The bulk of the embedded developers right now are aging boomers who are soon to be exiting the workforce. I barely know anyone else my age who does this stuff.
Yes the ecosystem is changing. There's been a "recent" movement towards ARM-all-the-things, even the little bit-banging chips that everyone is talking about. You can get uCs with clock rates from the single digit MHz all the way to GHz with loads of features. Most people seem to be running an RTOS on top of anything over a few Mhz but there are a few of us who do things the hard way when things get tight.
That's just the uC stuff though. There's so much more going on in the industry. Someone needs to write the windows drivers for your system, same with pretty much everything that's going on in the linux kernel, android, etc. Embedded wraps the entire computing space. Can't really say the same for any other discipline.
On top of all of this, there's a solid demand from the "mission critical" world for good devs. Airliners, satellites, rockets and cars all, by necessity, run extremely thin stacks consisting almost entirely of C (though some of us are starting the push for Rust). You can't just Python your way to glory in a fly-by-wire controller because that's how you get people killed.
Yes, the pay is mostly shit unless you work for a large company. Qualcomm and similar typically offers six figures starting. I work in aerospace but they're starting to get desperate as they no longer get to pay nothing for getting to build cool flying stuff. YMMV here.
Fellow late 20-something mobile turned server turned embedded dev here as well.
It is indeed bullshit.
I'm working on my Master's in CS right now, and my stepdad, a professional web dev who started in the Apple II days, used to ask me "Why do you still want to learn low level code/hardware/ways of doing things? It seems the industry keeps moving towards higher level frameworks/hardware/products."
The answer I've always given him is because someone has to build the world that the easier to understand, higher level pieces live in. There's still a niche for bespoke, highly performant and well architectured low level code, and there's not really a shortage of jobs because it's not what's being taught in undergrad CS programs, at least at my alma mater and others in my area.
> "Why do you still want to learn low level code/hardware/ways of doing things? It seems the industry keeps moving towards higher level frameworks/hardware/products."
> ... someone has to build the world that the easier to understand, higher level pieces live in."
Ah I love reading that! Some people really enjoy watching other people use or get use out of something they built or helped build. Some people just like knowing that the things they built or worked on are useful to folks and make their days / lives a bit easier. Then there are folks like you who are content to know that exactly because nobody knows your work exists or knows that you worked on that work specifically pretty much means you've done a great job. I'm sure that's an entirely different type of good feeling of knowing the things you worked on were useful.
IMO the reason to do embedded is it's super cool. You can actually hold something up and say "I made this". I don't really care for web because it's so ephemeral. I want to be like the previous generation that could point up at the sky at Hubble and say "My code is running on that".
Unless you really love doing low-level stuff, I'd still probably recommend you don't though. As a career, it just doesn't give you the same opportunities.
Yes, there are jobs, but I doubt you'll find the scope for advancement, and mobility that being a skilled dev (with good bit manipulation/low level skills) would give you.
An SRE role at Google might be a good example of a position that would involve a good deal of low-level debugging, but is still paid well, looks great on a CV and could advance you to other (and equally interesting positions) easily.
I actually find the opposite to be true. I can spin up containers, write ruby backends and do frontend design like anyone else. Moving up the stack is easy, especially when you know a little something about the performance impact of doing this vs that. I've seen a lot of people try and fail to move down though. It's a very complicated world that requires a lot of time to get comfortable in. Yea, driving peripherals is "just" writing to a particular memory address but where do you start? How do you find all the relevant documentation? What are the unwritten conventions of how that's done? Big endian/little endian? How long can I make this Interrupt Service Routine before things start to break?
Yes, I agree technically it isn't difficult. Depending on your location career mobility maybe less easy if you get pigeon holed as an "embedded engineer".
If you're doing it because you enjoy it, that's great. But ultimately embedded development doesn't have has good prospects career-wise.
I agree with this. Did embedded for years. Actually started out doing device drivers—as far as I know my name is still on a little driver .c file in the Linux kernel, my one meager claim to fame in life LOL. I eventually left the field partially because of the nonexistent career growth curve. Kind of sad that you can make 2-3X what a typical embedded programmer makes by barely holding some front end JS together, but that’s the reality of supply and demand.
Yeah that is not true at all. I've been doing embedded work for about 8 years and regularly get recruiting messages from/interviews with Apple, Amazon, and Google along with a bunch of smaller companies from around the US. I am currently at a startup working on autonomous quad-copters and we basically can't find really talented embedded engineers. Between IoT and autonomous systems the market for embedded developers is great right now.
Depends on location perhaps? But personally I've not seen that. Google etc. will always be looking for all kinds of people. But are they looking for as many embedded devs? Are they senior positions? If you want to progress within those companies will you need to move out of embedded dev? Those are the questions I'd ask I guess.
Unless you love doing X stuff, I'd recommend that you don't do X stuff because people should ideally do what they love and avoid doing what they don't love.
What I mean is there are aspects of embedded work that can be found in other jobs which are potentially more lucrative and may make your life easier later in your career. So, unless there's something in embedded work in particular, that you can't find elsewhere, it might be considering those jobs.
An example might be, if you like bit twiddling, digging into hard often hardware related problem, you might consider something like an SRE role.
I say turned, when more specifically I'm just a jack of all trades at a small software shop that does custom work across all three domains. I do truly love it, and I'm using it to segue into some EE experience as well, but the majority of my work is still in the mobile side, albeit interacting with the firmware dev I do (lots of BLE stuff).
That sounds like a really nice mix of work. Personally that sounds like great fun. It's also giving you a skill set that you could transition into starting your own thing (building your own product prototypes if you get enough EE stuff). Which sounds great!
I was wondering if you could give me some information about being a embedded software engineer and the skill set required for it. It is something that has always appealed to me as a job. I just finished freshman year of an ECE program (mainly the computer engineering aspect). I have been doing side projects in C for a few years and more recently have been doing a lot of rust. I plan on doing a minor in CS.
Do you think that continuing on this path will still leave open the opportunity for doing embedded software development? The idea of doing safety critical engineering in rust really appeals to me but I am also interested in things like computer architecture so I don't want to only do only a CS degree. Thanks for any information.
ECE/CS minor is pretty much perfect for a super low level role. I was straight CS and I kinda wish I had more formal training in some of the electrical stuff. If you like it, try to get comfortable in VHDL or Verilog FPGA stuff. That opens a lot of doors as well. Being able to do everything from board design through drivers will make you popular with smaller companies and startups that need that capability.
As far as employment, there's a lot of stuff going on. Traditionally aerospace (where you'll see the super hard-real-time mission critical stuff) is a PITA because they know the work is desirable and can therefore pay peanuts for people to do it. That's changing though so keep your eyes open. Also, don't do anything that could jeopardize your ability to get and retain a security clearance if you're US based. You'll need one for a lot of those kinds of projects (though not all).
Finally, get used to process. If you do critical work, the more critical, the more process you have to follow. Lots of code reviews, heavy handed version control rules, design and deployment paperwork, etc. It's not bad but it seems to be a shock to everyone.
If you have specific questions, feel free to contact me at gmail.com. I have a couple other responses in this thread that might also be useful to you.
This year I took a logic design course that had a few labs in VHDL with FPGAs which was a lot of fun. Thanks for the info, its good to know that I am on the right track.
I have heard a little bit about the process from a friend of mine who works in aerospace software engineering, and according to him like 90% of his job is design documents and other paperwork.
I am kinda surprised to hear that the pay isn't very good, but its good to hear that its changing. Is this mostly due to the older workforce retiring or are there other factors that are causing it?
Keep in mind that "pay" is a very nebulous concept in software. Embedded work clusters around the same cities that high level work does but I've found there to be more variety in some aspects. Defence work in particular will take you outside the bay area so that helps immensely in terms of income vs expenses. Small companies I've found don't pay well though. Everyone is talking about supply and demand but the reality of the situation is there's a second factor at play. Devs cost money and make money for the company. In web land, you Ctrl-C/Ctrl-V every time you want more money. In the embedded world you're usually interfacing with hardware and hardware has a monetary cost to produce. That cost is actually insanely high by tech world standards so that kills the massive margins that would otherwise go towards fighting for talent. There's also typically more risk due to the fact that you can't just edit a production application to fix it. You have to re-run production every time a flaw is found in hardware and sometimes even in software. That also moves money away from talent.
Now, the advent of cheap chinese manufacturing and increases in R&D cash help with a lot of that. The consumer electronics world is cutthroat but research work is pretty easy to come by and IMO very rewarding. Space seems like a potentially profitable sector at the moment, though that currently hinges entirely on one guy and his rockets. If it continues to take off, expect a gold rush in a decade or so.
Long story short, you'll still make good money, just not the 200k starting salaries of the bay area for the most part. Do what interests you and makes you happy or you'll burn out.
>Airliners, satellites, rockets and cars all, by necessity, run extremely thin stacks consisting almost entirely of C (though some of us are starting the push for Rust). You can't just Python your way to glory in a fly-by-wire controller because that's how you get people killed.
A friend mentioned to me that Spire is doing satellite control software in Python. Presumably it's higher quality than your average python project since you can't wipe it and provision a new one on a fresh satellite when something goes wrong.
Some satellites do run higher level code on the payload side. My company also uses it for our open source telemetry package. That code is typically very isolated though. If it crashes entirely, you can't exactly send someone over to hit the reset button or re-upload the image so you need some sort of hardened supervisory code running on radiation tolerant hardware. The big gotcha though is that space-capable hardware is frequently at least a generation behind commodity stuff so you don't really have a lot of overhead to deal with interpreters or VMs. This is less of an issue on cube sats where you don't care (as much) if it doesn't work and the mission is very short and non-critical.
On the bus side (the part that controls the satellite's support systems, thrusters, etc) having garbage collection kick off while making orbital adjustments is no bueno. Don't really have to say much more than that.
Not sure if that’s your job req or whether you just found it online, but here’s some unsolicited feedback on it: the way the qualifications are written there are probably less than 10 people on earth who meet them all. As someone who has embedded experience and an interest in space tech, if I were in the job market I’d be so intimidated by that description that I’d probably not even bother applying!
Pro tip: find someone willing to mentor you. Look for companies with people who are willing to help you with that, some will, some won't.
In the low power realm there's a lot of domain knowledge to learn. What sleep modes allow what, when and how to use them, etc. Don't give up though. It's very rewarding.
> The bulk of the embedded developers right now are aging boomers who are soon to be exiting the workforce. I barely know anyone else my age who does this stuff.
Just for some context, a decade ago I was saying the same thing. Glad to see _that_ hasn't changed in the meantime. It's also a core part of my motivations for having _left_ embedded in the first place.
Yea, obvious slant that makes about zero sense.. embedded devices firmware devs are worth a ton if they can quickly figure out how to use low cost chips to get a lot done. Maybe in the authors case he is like: “Step1: buy 450MIPS processor with 1GB DDR, step2: install API abstraction layer, step3: sell product for $200 to monitor your water pressure in your house”.. then sure that maybe is a thing.. but the reality of making products is, oh hey can I use this poorly documented 25 cent processor that has an adc and some gpios and needs a custom bit banged protocol to talk to a special purpose sensor chip.. yea so where’s the magic api for that???
I don't think his take makes much sense either. But you can get a MCU with embedded wifi/tcpip and 80MHz that costs about 1USD (esp8266). There are also pretty cheap/low power full Linux systems (KeyASIC make one that sits in an SDCard, similar to the eyeFI). They're not too expensive either (~10USD).
The ESP8266 is sort of a unicorn chip.. prior to that chip existing virtually everything in that space was $10 for getting wifi bolted onto a design unless you were a super high volume company who could get Broadcom's attention. In fact, the Electric Imp which was the SDcard / wifi combo thing was actually designed by somebody who left apple and managed to actually get broadcom to sell them chips -- I'm fairly certain they just happened to still remember how to use the broadcom chips that virtually nobody in the world has outside of samsung and apple and the associated SDK's -- so when that tiny electric imp startup used that chip I was optimistic that broadcom would make their chips more avail, but nope.. no change, and finally ESP8266 came out and changed the market completely for wifi.. pretty amazing thing..
To add context (and as you probably know). The esp8266 is an older design repurposed as an "IoT" play.
It was actually originally designed as a wifi interface. I think it was mostly used in cheap Android tablets. It just so happens that any generic wifi interface IC has pretty much everything required to make a general purpose IoT microcontroller.
That is to say, they've all cost an ARM or other cheap core inside as well as the Wifi physical interface. Usually the CPU just does data formatting, interfacing and other management tasks.
So, there are lots of those around. In fact if you search for USB wifi dongles on ebay you'll find them for 1 or 2USD. They all have some similar controller inside.
The difference with the esp8266 is that they released the documentation, and started marketing the chip as an IoT device. It's neat, because essentially all the of development costs have already been paid off by the Android tablet market.
And they've since released the ESP32 which adds a second CPU core for dedicated application code, as well as Bluetooth support and many other hardware peripherals. All for a few dollars. It's quite an amazing controller.
Possibly relevant to the discussion: The ESP8266 is capable of running REPLs for JavaScript, Lua, Basic or MicroPython, which probably makes development a whole lot easier (cheaper?) if the project isn't considered to be very critical.
I've done embedded work. I could do embedded work, I probably wont because the pay is mostly terrible compared with either traditional software engineering or the more data orientated stuff I usually do.
So, if there is a shortage of embedded engineers, it's because nobody really want to pay them properly. As to moving things up the stack... well that's always been a trend. But someone has to write the board support package and do all the hardware interfacing to the often broken and buggy hardware.
I think the pay is terrible because this type of work is now mostly being done in parts of the world where labour costs are lower than the US.
For example, I was working with a device that came with a SoC from a large American/global SoC manufacturer. I ended up disassembling some of the firmware blobs and was able to find the devs on LinkedIn who wrote the BSP and the power management firmware. I found large networks of teams of devs in India working on these very esoteric things.
I think the pay is terrible because this type of work is now mostly being done in parts of the world where labour costs are lower than the US.
This is true, and it works for small localized code...but when it comes to application work it backfires a good amount of the time. A serious portion of embedded application work is digesting the business knowledge of the client and translating it into working firmware.
If you need to take a stateside engineer and write and review the specifications to the code, then throw it over the wall, THEN review the finished work before delivery....what was the point in the first place?
I always found that bemusing. The more low level, difficult and involved your programming job the less you get paid. It makes sense on a capitalist pays for productivity level but it really doesn't make sense on any other level.
It makes sense on all levels. Pay is not correlated with how hard your job is. It's correlated with how large of an impact you have on the business.
It's really easy to have $1M+ (or basically very large) impact on a business by making data more accessible and providing solid analysis.
On the flip side, it's really hard to have a really large impact writing low level embedded code.
The reason for that is because since it is a well defined problem (control this hardware, send data from here to there) there is not a lot of room for creative flexibility. So it's hard to differentiate yourself above others (even though very technical people know that the quality of low level code can vary dramatically, this often isn't reflected in the success of the business).
At the end of the day I always ask myself, is what I am doing driving business value for my clients. If not, I try to reframe what I am working on so that is the case.
> even though very technical people know that the quality of low level code can vary dramatically, this often isn't reflected in the success of the business
I would rephrase that as the quality of low level code is not often recognized as having a huge impact on the success of the business but when low level code goes wrong it can have very costly negative effects which ripple up through the company. The usual manifestation is a buggy product with many layers of hacked together bloat. Low level code often does not get the attention it deserves because the higher ups don't understand it.
You are missing my point. The point is that the impact one has writing embedded software is largely defined in the spec. Other than perfectly meeting that spec, there is not a lot of room to go above and beyond and create extra value. I'll admit I am not a 30 year veteran in embedded development, but I did spend a few years in the field.
Embedded development is very time consuming because it's generally so critical to get right and keep bugs to an absolute minimum. Which is also why it's important to keep it as simple as possible. This inherently limits how creative one can get.
I suspect its because the work itself is more fun for a certain kind of people than pushing data from A to B. You also need more data pushers than bit-bangers, so maybe there is not such a shortage?
OK, then we will call them driver writers and they will do exactly the same stuff. An embedded application that doesn't do something fairly unique with the hardware isn't an embedded application. It's just a regular computer application. That's why we make the distinction.
Low resource systems won't be going away any time soon either. The ignition (or battery controller) in something like a weed whacker is unlikely to be communicating with anything and will likely be running on a 8 bit processor with no room for libraries.
I don't see that the problem is that we will lose the skillset as suggested in the article. The bigger problem is that we have trouble producing embedded programmers in the first place. The universities/colleges are doing a terrible job. Most embedded programmers picked it up on their own by messing around.
Well that's just silly. Is your CPU going to work with bespoke hardware connected on the whatever bus? Then you still need to know both software and hardware to debug the connection to the bespoke device. Stuff rarely works the first time; FPGA designers make simple typos just like software engineers do.
Yes, the ecosystem is changing. Instead of a wide range of embedded CPUs, we're shifting to tiny bit-bangers (8051 or Arduino) and most everything else is ARM Linux. CPUs were never the hard part of embedded. The hard parts have always been (1) debugging the hw/sw interface and (2) meeting realtime requirements, if the system has any.
Embedded people will be extinct because of the salaries. Embedded SW I wrote runs on hundreds of millions of devices and anything from a single core small ARM to a multi-architecture SoC, but when I was last looking for a job--higher in the stack things like infrastructure / DevOps paid 50% more than embedded.
Yeah, it's really sad. I've dabbled in embedded stuff, in the course of doing lower-level OS/infra stuff. The people I've met who make a living at it have generally been in the top quartile for development skill, because the consequences for doing a sub-par job are severe and that naturally winnows out anyone further down the scale. For a long time pay scales did reflect that, but not in the days of front-end types (who call themselves "full stack" without a trace of awareness of how deep the stack really is) and ML/AI folks. I know it's all about supply and demand, but I think "demand" is being mismeasured in this case. People don't think they need a good embedded developer, until they really really do.
Are you kidding me? We need these guys like you wouldn’t believe. Does anyone really trust that their bios / drivers are actually secure??
My job is to secure Web Applications, so usually
cleaning up PHP code. But literally everything I do is pointless if the lower level firmware is compromised (it probably already has been).
Very high demand for embedded engineers => too expensive, hard and time consuming to train the needed engineers => market responds by creating API’s so less “traditional” skill is required
You know what you call embedded software developers who move up the stack and start programming in Java and “don’t understand why the hardware works”? Not embedded engineers.
I guess I'm working lower than embedded, then. For the things I work on, there is no API, just a chip data sheet, and usually it is marked "preliminary" or "advance copy".
Hahaha, I've absolutely learned a little written Chinese for reading datasheets. It's not that bad either, resistor for instance is "electricity pushing back device".
Wow, massive Gell-Manning effect going on in these comments.
I've taken one of Jacob Beningo's courses after working for a long time in embedded and I know exactly where he's coming from. Granted the title is a bit clickbait.
The fact that he's "selling something" doesn't mean he's shill. It actually underlines his authority on the matter.
In this short article he only had a short space to address the state of embedded engineering. But of the key trends, IoS is only one of them. The other important one is the abundance of CPU cycles. It's hardly required to be close to the metal these days. In some cases, yes, but things are hardly bound like they were 10-20 years ago.
Also, perihperals are standardizing. This means the skills are more abundant. Plus, education is widespread, so many things that were mysterious aren't anymore. Next, product cycles are shorter, so shortcomings in products can be fixed in the next rev when better HW is available. Also, SW engineering practices are getting inherited from the rest of SW eng industry, so quality and predictability are increasing. Also traditional CS people can be very effective in embedded areas.
However, there will always be about 10% of the market that is difficult enough or requires enough expertise that only highly experienced people can achieve. These embedded engs can't be replaced any time soon.
Someone still needs to implement the frameworks that these so-called embedded engineers will be using! The article is right in that users of embedded devices now expect a lot more software support. I work for a semiconductor manufacturer supplying devices to tier 1 auto electronics companies and these customers expect a full OS port with drivers and middleware for the device (including a lot of graphics processing for ADAS applications).
While I get the author is tired of people who have mastered the use of Arduino libraries calling themselves "embedded programmers", that additional capability is available which opens up embedded programming to more people isn't a bad thing. It will be incumbent on people who hire them for jobs involving Fire & Life Safety (FLS) systems to recognize their limitations.
Seems like the opposite, there's a smaller revolution going on in audio and music tech. Several big software names are about to start manufacturing their own dedicated hardware devices running DAWs.
In Japan, with the possible exception of certain industries -- such as video games -- programming historically hasn't been seen as a professional practice in its own right, but something done in the course of other engineering work. I think this is where embedded software development is headed. The real low-level stuff will be done by HWEs; everything above that will have an easy peasy Node wrapper that you can assign Fullstack Academy grads to develop for.
This has been the push since the dawn of computing. The need for low-level engineers is much less than need for high-level engineers. This is not just the case for embedded developers, but also for OS, database, browser, IDE, etc. developers. A handful of people who are experts in these fields support the millions of "high-level" developers that build apps.
I'm sure most of the people in these fields have generic degrees in CS/CE/EE.
I've recently transitioned from being an "application developer" (to use the article's terminology) to embedded firmware.
The difference between the two is not as great as the article is trying to make it. Yes, there are specialized skills, but that is true of any platform. Win32 and iOS are as far apart from each other as iOS and embedded.
I've worked with several developers with 20+ years embedded experience who wrote horrible, unmaintainable code.
However, the article's main tenet I agree with: Higher level abstractions are being pushed for lots of platforms. A great example of this is Google Things [0]. I have my doubts about the quality of product you'll get by asking Android developers to write "firmware" for you hardware product, but it is evidence of the general push. The author seems to accept that these pushes will be immediately successful. I have my doubts.
I agree with a few comments here about how the field seems to skew heavily towards the older range, but based on my job hunt in NYC there are very few jobs like this.
After a week and a half I have no interviews even though I have a few beefy side projects (embedded Linux system I designed myself), use both C and c++, heavily, contributed a bug fix to the Linux kernel, and even have 3 years experience at a company. To be fair, there are few jobs like this it seems on nyc (managed to find only ~25 on LinkedIn).
To all those who say that we will always need Embedded Engineers to keep things optimized at the register level,
I humbly put forth these two research, which may possibly automate optimization at the hardware level
As someone who held the title of Embedded Systems / Embedded Software engineer at multiple places, just want to respond that optimizing things at the register level is NOT the main focus, or even a sizable amount of what embedded engineering is all about...
> Interestingly enough, microcontroller manufacturers are currently in a big push to provide developers with high-level software frameworks and tools that abstract out the low-level hardware.
Ha. I wish.
We're usually working with chips where the software from the chip vendor isn't done, and isn't going to be done before we need to ship a product.
And then there's issues where the vendor calls it 'done', and it isn't done, and we have to fill in the gaps.
This article doesn't ring true for me at all. I'm a recent (last year) university grad from a "software engineering" program that was super heavily focused on electrical engineering and low-level embedded stuff. I currently work as an embedded software engineer. Maybe it's the only program like this left, but I doubt that. Where does the author back up his claims that universities are shifting away from offering this?
This is horseshit.
The bulk of the embedded developers right now are aging boomers who are soon to be exiting the workforce. I barely know anyone else my age who does this stuff.
Yes the ecosystem is changing. There's been a "recent" movement towards ARM-all-the-things, even the little bit-banging chips that everyone is talking about. You can get uCs with clock rates from the single digit MHz all the way to GHz with loads of features. Most people seem to be running an RTOS on top of anything over a few Mhz but there are a few of us who do things the hard way when things get tight.
That's just the uC stuff though. There's so much more going on in the industry. Someone needs to write the windows drivers for your system, same with pretty much everything that's going on in the linux kernel, android, etc. Embedded wraps the entire computing space. Can't really say the same for any other discipline.
On top of all of this, there's a solid demand from the "mission critical" world for good devs. Airliners, satellites, rockets and cars all, by necessity, run extremely thin stacks consisting almost entirely of C (though some of us are starting the push for Rust). You can't just Python your way to glory in a fly-by-wire controller because that's how you get people killed.
Yes, the pay is mostly shit unless you work for a large company. Qualcomm and similar typically offers six figures starting. I work in aerospace but they're starting to get desperate as they no longer get to pay nothing for getting to build cool flying stuff. YMMV here.