In my opinion Toyota firmware engineers were not empowered check the quality of their own firmware, and were likely under time pressure to just Get-It-Done™
I've managed embedded C & C++ products in my career, and I always insisted that the developers be given the time to set up the firmware to run under representative load in a lab. Without fail the developers found significant problems that did not crop up during QA or dev unit testing. Tasks performed by the devs included creating and evolving simulation hardware, firmware and software to make the most representative load scenario possible to find and kill chaos. The funniest part? Nothing I ever did was in safety critical systems, and we clearly did more than Toyota.
Page 31: References Toyota witness who indicated that they didn't think the firmware had bugs
Page 53: Critical variables were not mirrored!
Page 56: Throttle could be thrown wide open by memory corruption bug
*Page 62: Task X (real task name redacted) in the code called out as a "Kitchen Sink" task that does a bunch of different things, including cruise control and throttle control, and, most importantly, reporting diagnostic errors*
Page 64: There was no brake override to cut throttle when brake applied. Not only that, but later Toyota model threw brake override code into the kitchen sink task anyway, so if that task failed.....
Page 82: Barr describes what happens when task X is killed after user hits resume on cruise control
Page 89: Additional failure condition, if use is lightly touching brake when cruise control resumes and task x is killed, the acceleration won't stop unless they fully release and press break pedal again
Page 91: Buffer overflow found in code
Page 92: Pointers dereferenced without checking to see if they are valid
Page 93: Nice description of a race condition
Page 99: Toyota employees referred to the code as "spaghetti"
*Page 100: Code used over 10,000 global variables*
*Page 107: Stack size is 4kb, Toyota thought it was at worst case 41 percent used. Barr found it to be 94 percent worst case.....not counting recursion that was in the code as well!*
*Page 108: In case of stack overflow, critical OS data, such as what tasks are running, is the first thing to get overwritten*
Page 112: Toyota engineers didn't understand that the stack also gets data pushed onto it by ISR and OS task switching
Page 113: Toyota code had recursion, a huge no-no in critical safety systems
Page 123: Toyota engineer indicated they never looked for a firmware bug that would cause the throttle to be wide open
Page 128: Mirroring was used on many variables, however the target throttle angle variable was NOT mirrored
Page 138: RPM failsafe was part of Task X (Kitchen Sink task)
Page 142: Watchdog only monitored for excessive CPU usage. It didn't check for tasks being killed.
Page 143: Toyota dedicated a hardware timer to kicking the watchdog!
I've managed embedded C & C++ products in my career, and I always insisted that the developers be given the time to set up the firmware to run under representative load in a lab. Without fail the developers found significant problems that did not crop up during QA or dev unit testing. Tasks performed by the devs included creating and evolving simulation hardware, firmware and software to make the most representative load scenario possible to find and kill chaos. The funniest part? Nothing I ever did was in safety critical systems, and we clearly did more than Toyota.