Having spent the better part of 30 years working on/with/around embedded systems, I can't even count how many bugs I've bumped into that were hiding inbetween sofware and hardware. Or between software and compiler/tools/OS. Or between hardware and spooky RF black magic.
I got one. Embedded linux board with a radio transceiver that would occasionally crash when the radio sent a packet out.
Found two problems.
The reset push button was acting like an antenna. Caused the reset chip to assert reset. Confusingly the reset pin was bidirectional so hard to tell who was asserting it. 10k pullup fixed that.
But the problem persisted at a much reduced level.
Finally figured out the RF was tripping the over current protection on the low drop out regulator for the processors 1.1V core supply. Voltage would sag about 50mV which was enough to send the uP into the weeds.
The first one was actually a surprise since often you don't see that. Most random stuff are really really bad crap antenna's.
The second one is pretty simple. The voltage regulator has a internal circuit to detect over current and turn off the regulator. So if current is too high or the part gets too hot it turns it off preventing the regulator from letting the smoke out. In this case somehow the RF was tripping that circuit for about 50ms.
