Toyota had more than 10,000 global variables.
“And in practice, five, ten, okay, fine. 10,000, no, we're done.
It is not safe, and I don't need to see all 10,000 global
variables to know that that is a problem,” Koopman testified.
and:
Toyota’s failure to check the source code of its second CPU,
supplied by Denso —even as executives assured Congress and
NHTSA that the cause of UA couldn’t be in the engine software
and:
He was critical of Toyota watchdog supervisor – software to
detect the death of a task -- design. He testified that Toyota’s
watchdog supervisor “is incapable of ever detecting the death
of a major task. That's its whole job. It doesn't do it.
Instead, Toyota designed it to monitor CPU overload, and,
Barr testified: “it doesn't even do that right.
and:
Barr also testified that Toyota’s software threw away error codes
from the operating system, ignoring codes identifying a problem with
a task.
When the news first broke a few years ago, given Toyota's reputation for quality and process, I thought this was an American industry lead witch-hunt of a Japanese competitor. But if this testimony is correct, what Toyota engineers have done is unforgivable.
> “And in practice, five, ten, okay, fine. 10,000, no, we're done. It is not safe, and I don't need to see all 10,000 global variables to know that that is a problem,”
At an old job, my boss asked me to take a look at a business critical application to see if it could be improved upon. It had some deficiencies that were really hampering things.
I got the source from a coworker (this is when I worked on rockets, so none of us were or are professional software people, but even my dumb engineer self knows this is not how it should be). I opened up the folder. Lots and lots of files with little apparent structure. Ah, there was a file called "MAIN". I opened it. Visual Basic 6. Over 29,000 lines for global variable declarations alone. The actual program logic heavily used a commercial library designed to (shudder) make VB6 development more like making macros in spreadsheets. The original programmer then implemented the main program logic in this hellish abomination. The logic took up another 50,000 lines or so.
I told the boss that it would be easier to write a new program from scratch then attempt to understand it.
Has anyone seen a program with more globals than this?
I once worked for a billion dollar retail company. Their core back office applications (sales tracking, logistics, orders and fulfillment, staff rosters, etc) were written in COBOL and the inputs and outputs were read from / written to temporary files; kind of like pipes but via temp files in case of job failure. The real problem was that there were about 3,500 COBOL programmes and 2,500 shell scripts all in one directory. All the executables, COBOL and shell alike, were given a 4 digit filename; there was no up to date documentation on what program did what (apart from the source), or in which order they should be run, or the dependencies there in. To top it off end of day processing would take between 12 to 16 hours to run; any major problems and it would start affecting the next day's EOD. There was only one guy in the office (who had been there 15 years) who understood how it all fit together; when he died things got much worse (no he wasn't that old, how he died is another whole war story all together).
I only lasted there 6 months before I left for a start up, in many ways it was too much to bear. A while after I left taxation for retail changed substantially, there was a massive IT stuff up and the billion dollar business wound up being carved up and sold off; several thousand jobs affected, many lost their jobs either directly or indirectly as a result.
Sure this is probably at the worst end of the scale (short of causing deaths) and is rare but it does happen. Billion dollar company succumbs largely because of IT failures. It got to the point that the cost of the technical debt was higher than the carve up losses...
To the grand parent: you're not supposed to look at generated code, if you have changes you're supposed to re-generate it. So at some level, that's the 'why' that happens.
In Cobol 85, all variable are global to the program, and statically allocated at program start up. It's not uncommon for a Cobol program to import 10's or 100's of 'copybooks' and have 100-1000's variables in the global scope. They are, however, name spaced after a fashion.
My experience with this came from a similar behemoth mainframe app; 35 MLoc, 15,000 programs, thousands of jobs, running a batch and interactive green screen app that a company hosted. Was the source of about $500M a year in revenue.
>To the grand parent: you're not supposed to look at generated code, if you have changes you're supposed to re-generate it.
Generated code? What do you mean? (I have no familiarity with VB6 at all. I have written quite a bit of code in C, Python, and MATLAB). Besides, all this was handwritten (I eventually had a conversation with the code's original author while at a party). The guy was crazy enough to write all of that by hand. I genuinely think the effort made him a little unhinged.
It is. Mozmill is a testing framework. What we're looking at there is a file containing automated tests (which I think should be fairly obvious to someone looking at that file...)
I guess we'll have to disagree, especially about this part:
> horror beyond comprehension
"Horror"? No, but I guess that's debateable (apparently). But "beyond comprehension"? Absolutely no. Even if we say the repetition offends our aesthetics, I can't even imagine what sort of issue you had to deal with where this file posed a problem. Anything you could possibly want to do with it should be serviceable inside of 5 minutes, except in the dumbest of Notepad-quality (i.e., little more than a text control) editors. That includes rewriting the whole thing to be less offensive, if that's your slant.
This file is nothing compared the types of things mentioned upthread, things which, again, exist in a context of actual production code.
That's not bad at all - it's a lot, it's not DRY and it could be reduced to about 25 lines of code (or 25 + the number of URLs to check), but it's legible, clear, and easy to understand.
It's a strange feeling when some small webdev are using clean modular testable code while real life businesses are abusing VB in creative ways. Really makes me feel paradigms do matter. My last job involved a huge pile of VB (VBA to be exact) code and again, it was an unstructured mess.
After having lived and worked in Japan, this doesn't surprise me. Software engineering education is really weak there, managers who became managers because of their seniority tend to impose their own antiquated decisions and do not listen to best practices. The other problem is that status wise, it's considered lower level to be a software engineer than to be a mechanical or electrical engineer, so the smartest kids do not study computer science (this might have changed since).
This is also an interesting contrast to the (I assume higher quality) kind of work being put out by their other engineering departments- how can they make other sound engineering decisions on the rest of the car and it's manufacture, but fail at the software?
Is it something specific to software vs. traditional engineering in Japanese culture, or is it something inherent to software in general? Given the parent's complaints about workplace culture in general, shouldn't the entire car be badly designed?
I worked in new auto development at Toyota (Japan) for four years in mid 2000s, right before this happened. The unintended acceleration problem has also perplexed me, because in my experience any safety issue was always given top priority. But I did experience limited familiarity with software/firmware. Usually that was looked at as part of the component, for example the engine control module in it's physical plastic housing. Remember the production auto will have Denso (or similar) delivering that ECU to the factory as an "assembly" and hundreds or thousands of these are assembled into the car and function together. At the end of the assembly line, various tests are run on the system and vehicle, but these appear to be of the "input/output" that someone else described. So why isn't software done better?
I'd say one problem is the testing process is not robust enough for software/firmware as it is for mechanicals. The tests revolve around driving the car, and variations on driving, and have evolved over years and decades. Heavy software control is newer, and in my experience was often not fully understood. Chief Engineers tend to come from mechanical. And there are always hundreds of problems to solve and limited time, and the black-box nature of firmware makes it hard to develop tests for.
A development review meeting tends include various mechanical departments, like Drivetrain or Engine or Chassis, many times they bring physical prototypes to the meeting and the problems and proposed solutions are comprehensible. The build testing procedures subject the car/parts to various mechanical tests, and long-lead tests are performed which stress parts till breaking. It's something you develop experience with an can get your head around.
Firmware bugs are not like that -- having just gone through this with my current consumer electronics startup I experienced the black box component first hand. I remember one software based problem we reviewed at Toyota, where something happened like: when the car has been sitting for >3 hours at <50F weather, and the ignition is started when the car is at a 15 degree angle (side of road) and driven on to a flat plane within 15 seconds, the "check engine" light turns on. Even though we had no test for that, we heard about it for customer complaints came through. But the fix is to have the supplier (in this case, but could be the Toyota engineer) fix that problem, with most of the discussion centered on timing and certainty of fix. I don't remember any discussions about global variables or lines of code.
my perspective from some friends that bought a Japanese software startup and moved over there to help run it a few years ago:
promotion is based on age and length of employment. employment is essentially for life. you can't really be fired and you if you quit you may never again be hired (and i can't imagine whistle blowing would bring you any better). seniority reins supreme and unquestioned. there is a huge focus on hard work and long hours but also on loyalty and face.
contrast this to where and how you're working and think about the differences it would make. my friends were constantly frustrated that, try as they may, they could not elicit feed/push-back from anyone that worked under them. part of this is them being stupid foreigners that can't "read the air" properly, but part of that is people being extremely hesitant to touch anything that could be considered a break in seniority or disloyal. and this was a start-up. these were the f'in cowboys in comparison.
from what i gather, this system worked out pretty good with the long, patient and deliberate development cycles in hardware. when a product cycle takes ten years it might not be that bad to have people running it 'cause they've been there 10-20 years. but it doesn't work very well in software.
Exactly why I'm commenting so much on this. I always tell my team "Pay attention to detail, quality, method. Be more like the Japanese", to now see that even the Japanese (and their quality poster-boy no less) have trouble applying their quality philosophy to software. Not sure where that leaves my advice.
It doesn't change your advice. Toyota has a well-deserved reputation for quality in a lot of areas: customer service (they've been replacing frames on trucks for a few years at no cost to the consumer), construction, reliability, maintainability. That they have flaws in one area just means that they need to apply their practices to that area too.
There was an article about this (in New Yorker, I think...) a few years back, t how Japan has this strange relationship with software.
How it is a hardware-first country. Many reasons were given -- cultural, how "real men build real things", infatuation with appliances, which are seen as being "hardware" in a way, perhaps the langauge barrier i.e. existing software tools using English for menus and inputs etc.
And it is kind of interesting, looking back at a country that many in the 80's feared with surpass and leave the Western world in the dust technology-wise, we haven't seen that much software produced there. Can you think of large well known software from Japan (excluding console games). Vine Linux, SoftEther VPN, TrendMicro, anything else?
One wonders if they see Western software companies and if there is any desire to catch up or try to keep up.
Maybe anyone from Japan can provide a better perspective?
I'm not from Japan, but I work in the US for a large Japanese company. I know that they have a large software engineering team in Japan, but if it isn't hardware related they seem to have a hard time producing quality software. They seem to have realized this, and now focus on producing hardware and have started up 3 different software teams across the US to build software for said hardware.
They occasionally send some of their engineers to our offices (for 6-12 months usually) to learn about our culture. There is a clear disconnect when they send these people. They work hard, no problem with long hours, but they have a very hard time getting into the collaborative nature of our teams, and I don't think it is just the language barrier. I guess I take it for granted usually, but it made me realize how much time we spend discussing things between developers, questioning management, writing furiously on the whiteboard, arguing, trying things out, failing, trying something different, finding problems with other people's code, fixing said problems, etc. It can take a bit of thick skin to absorb such criticism and engage in this process, but I think it is crucial to good software development. Someone who just sits quietly in meetings, then waits for a task to be assigned, and works at that task until it is complete is somewhat missing the process, because sometimes it turns out the task your manager assigned doesn't make any sense, or can't be done with this framework, or that you would have to use 100 global variables to accomplish the task this way.
I am merely speculating but it's possible that a very rigid social and management hierarchy where it is more important to work hard and respect your position in the company than to solve the problem the right way if it involves bucking authority, could negatively impact the ability of developers to innovate and/or produce a quality product.
That's certainly not a shining example of software quality. I also don't think it's fair to generalize about software quality for an entire country based on 1 or 2 examples.
I would say games are probably where Japan has most excelled in software, at least in the 80s and 90s. But the code is probably very bad practice by modern standards (hand coded assembly hacks, all game state in global variables). When you only have a few KB of RAM/ROM there is a lot less scope to mess up (although I have come across bugs in cartridge games myself).
The console frameworks and API's are all produced in an SDK that other developers use .. so its just as valid an example of a successful Japanese software product.
> it's considered lower level to be a software engineer than to be a mechanical or electrical engineer, so the smartest kids do not study computer science (this might have changed since).
Which is very interesting, because in US it is completely opposite (and frankly it is not good either). Is it because US deindustrialized itself last 40 years?
It was not the opposite in the US 30 years ago, when I chose EE over CS, because EE was clearly more respected as being the more intellectually demanding. This had a lot to do with the capabilities of hardware at the time and the locus of value. IBM sold its hardware for millions. It gave away the software. The hardware was the differentiator; software was generic. EEs who could improve the expensive hardware were a lot more valuable than programmers who could improve the free software. In fact, since doubling a hardware resource such as RAM far more than doubles what you can then build with software, the hardware guys were more important than the software guys--even to the software.
As hardware limits were relaxed over the years, what you could accomplish in software, even with cheap hardware, grew so enormously that the primary value was increasingly in the software. This was such a rapid transition in the computer industry that it nearly destroyed IBM, it propelled Microsoft to power, and I ended up with a career in software.
Toyota and other Asian "hardware" vendors are in some sense where IBM was. They sell prize-winning hardware and whatever software that comes with it is essentially free. (Apple is a lot like this, which is why their hardware keeps getting better and their software--well, did I mention the hardware is thinner?)
The Toyota "platform" still can't do much in software, but it's growing exponentially. Eventually the software in a car might matter more than the (commodity?) hardware, but that's not the world in which Toyota managers were formed. They are obsessive about hardware, but software is an afterthought. It's for guys who couldn't cut it as "real engineers", as was the case in the American "computer industry" until the 1980s.
I also lived and worked in Japan. I was a strategy consultant, and I was told by some big Japanese and Korean companies (whom everyone has heard of) to emphasize my hardware background and not be seen spending too much time hanging out with the software guys if I valued my reputation.
Apple is not like that at all. Their software is excellent. My guess is that a large percentage of their customers keep buying their hardware because of iOS or OS X not because it's thinner.
Did you mean "Deindustrialization or deindustrialisation is a process of social and economic change caused by the removal or reduction of industrial capacity or activity in a country or region, especially heavy industry or manufacturing industry."
?
That doesn't sound applicable to the USA, which currently produces more goods than ever before, and more than any other nation except China.
I can state that this matched my experience though I was in the US - this was in the 90s, and in the automotive industry. I used to work at Motorola Automotive and Industrial and our team built the Computer-Aided Manufacturing systems that managed the production lines. It always felt like a tier-2 job compared to process engineering.
Given how they had the researcher effectively locked up in a hotel room with strict security measures, I'd say it's because they're terrified of corporate espionage and suchlike.
I wouldn't be so quick to blame the engineers. They might be working within a managerial structure which doesn't give them the latitude they need to do their jobs. Yes engineers have a duty, but in software it's often hard to know exactly what the impact of cruft will be, and often all you can do is quit.
I agree, but I also believe that as an engineer I have an duty to refuse to build something under conditions that, once built, would cause harm to others. Or at least to relay my concerns to management in writing before proceeding.
And sometimes, quitting is the ethical thing to do.
Quitting certainly is the ethical thing to do. I wonder if I'd quit if I was in their position.
Handing your concerns to management in writing isn't really a done thing in this culture. I sometimes get laughed at for even suggesting it. And people who quit often take a massive pay cut, are forced into contract work with no safety, or even don't ever get hired in software again - especially in Nagoya where everyone seems to knows everyone.
And when you have a family and a mortgage to pay (selling a house here is also a guaranteed massive loss), it can realistically be a choice between you and your family on the street or doing what you're told.
While the engineers aren't blameless, the system over here is really broken.
But what if you have a life. And dependents. And bills to pay.
As much as I would like to agree with you, people have different value systems. And money talks. So quitting isn't always an option, even though it is the most ethical. Can't pay the bills with ethics.
> So quitting isn't always an option, even though it is the most ethical.
If you think quitting a dangerous project (to be replaced by another dime-a-dozen engineer by a large corporation which employs thousands) is the ethically correct choice, your moral compass is broken.
We're talking about a device sold in the millions of units that is capable of dealing great bodily and structural harm to us and our world around us.
Whistle blowing (preferably before real world harm) is the only ethical choice for someone that is made aware of such a situation.
Then you probably should not be taking on work that human lives depend on. Placing my bills over someone else's bills is a perfectly human reaction. But placing my bills over someone else's life -- a typical human should not feel this way.
Unless you're living without luxuries then you are prioritizing your access to luxuries over the lives of others. That's unconscionable, even if you word it as "bills to pay and mouths to feed".
If you're working as an engineer for Toyota's firmware it's unlikely that you won't be able to find sufficiently well paying work to keep the kids in clothes and food and keep a roof over your head.
And, at large Japanese companies, people are shuffled around every two years. I had heard this and didn't believe it until I saw it. A distributor of my company would replace people with novices once the previous person had just gotten really good at the job.
I was a victim of this. Thankfully only minor damage resulted. I owned a Toyota minivan in Thailand and was in stop-and-go traffic, when suddenly the engine started revving up. I slammed on the brake and the engine downshifted to first to compensate, so I ended up bumping the car in front of me. After the bump (total stop?) the engine recovered it's sanity I guess, because the revving stopped at that point.
Naysayers will point out that the engine wouldn't rev up if I'm putting on the brake. That seems to be part of the symptoms of this bug. Oh, and I didn't accidentally push the accelerator instead of the brake, otherwise I wouldn't have just "bumped" the car in front.
Edit: I took the vehicle into the dealership and they said "user error".... whatever, no consumer protection in Thailand anyway, so I just moved on.
The last company I worked for outsourced a large firmware project to Denso based on their relationship with Toyota.
Turns out they spent 18 months writing unmaintainable code that barely worked, and the codebase had to be totally scrapped with product version 2.0. The code was chock full of cut and past, globals, and hacked mutexes- they did not even use the mutexes built into the RTOS.
Perhaps Denso stuck their B-team on the project, who knows.
> When the news first broke a few years ago, given Toyota's reputation for quality and process, I thought this was an American industry lead witch-hunt of a Japanese competitor. But if this testimony is correct, what Toyota engineers have done is unforgivable.
These two possibilities aren't mutually exclusive.
Software engineering practices aside, was it ever demonstrated that unintended acceleration could actually arise from the behavior of this software?
Yes, another part of Barr's testimony was a series of proof-of-concept demonstrations on the ECU to demonstrate the ability to kill tasks, leaving the hardware those tasks were controlling in unintended states.
>was it ever demonstrated that unintended acceleration could actually arise from the behavior of this software?
That's entirely the wrong question to ask. The correct question is "has the system design been proven to make it impossible for unintended acceleration to occur?"
> When the news first broke a few years ago, given Toyota's reputation for quality and process, I thought this was an American industry lead witch-hunt of a Japanese competitor.
You. You were one of the many people I had no clue where they came from. When I first saw the reports they seemed legitimate and Toyota seemed to be hiding something but so many people just didn't believe it and the news just sort of went under the radar.
In C, there may be static variables with infinite lifetime which are locally scoped (module scope or function scope).
This is done to reduce stack usage; it does not mean that these 10000 variables can be accessed from everywhere else.
But I don't want to ruin the "hurr-durr, stupid C programmers" party that the rockstar full-stack webdevs here like to celebrate. Obviously, they know more about embedded software development than people who have worked in this field for several years.
The criticism of global variable count comes straight from Phillip Koopman, "a Carnegie Mellon University professor in computer engineering, a safety critical embedded systems specialist, authored a textbook, Better Embedded System Software".
So this isn't a web developer providing criticism, but someone with extensive experience in embedded software development. Perhaps reading the article linked before jumping to conclusions might be useful!
Locally-scoped (i.e. static) variables are, by definition, not global variables. The article is pretty clear that the 10,0000 figure refers to truly global variables, and I would hope that the original expert witness has not misled us by referring to static variables as 'global'. (Of course, without being able to see the source, there's no way to make our own judgement.)
Sadly, you would be surprised at the number of embedded systems that store and pass around their state using global variables, despite the obvious stupidity of that approach.
(For the record, I am not a web dev and I did indeed spend several years working in the field of embedded development.)
