Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Not a hardware bug, but in embedded I ran into a fun one early into my first job. I setup a CI pipeline that took a PR number and used it as the build number in a MAJOR.MINOR.BUILD scheme for our application code. CI pipeline done, everything worked hunky-dory for a while, project continued on. A few months later, our regression tests started failing seemingly randomly. A clue to the issue was closing the PR and opening a new one with the exact same changes would cause tests to pass. I don’t remember exactly what paths I went down in investigation, but the build number ended up being one of them. Taking the artifacts and testing them manually, build number 100 failed to boot and failed regression, build 101 passed. Every time. Our application was stored at (example) flash address 0x8008000 or something. The linker script stored the version information in the first few bytes so the bootloader could read the stored app version, then came the reset vector and some more static information before getting to the executable code. Well, it turns out the bootloader wasn’t reading the reset vector, it was jumping to the first address of the application flash and started executing the data. The firmware version at the beginning of the app was being executed as instructions. For many values of the firmware version, the instructions the data represented were just garbage ADD r0 to r1 or something, and the rest of the static data before getting to the first executable code also didn’t happen to have any side effects, but SOMETIMES the build number would be read as an instruction that would send the micro off into lala land, hard fault or some other illegal operation. Fixed the bootloader to dereference the reset vector as a pointer to a function and moved on!


Another 10/10 bug, thanks for sharing


From CI pipeline to bootloader would make me about turn and nope out of embedded so fast if that was my first job. That level of skill requirement is like a department in one. Hopefully that company had some patient seniors.


I think anytime you need to start stepping through assembly instructions instead of C in a debugger is also when you start reflecting on life choices.


I would be surprised if you have a full head of Hair after having dealt with that :-)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: