There's a good deal of responsibility in writing code that is going to run over every single page (and PDF document, MS Word doc, the higher ranked Flash animations, etc.) in the visible web.
For my part, I once made a bad assumption about how the Google SAX-style parser handled callbacks for zero-length XHTML start-stop tags. I presumed that <title/> would get a callback with the end-of-open-tag and start-of-close-tag pointers being equal, at the character after the close of the tag. Instead, the parser called the callback with the start-of-close-tag pointer after the start-of-close-tag pointer. (I had misinterpreted the API as passing pointers to the start and end of enclosed content.) I had test cases for un-closed <title> tags and <title></title>... but when my code hit production, the few pages (fewer than 1 in a million) that expressed an empty title as <title/> caused my code to try and construct a string with negative length and crashed that portion of the indexing system. I was right to feel very embarrassed for my oversight.
I remember the savage meme so clearly because it was quite out of the norm, and I felt bad for the guys since they were so quick to fix things even when not publicly shamed. (Only the author and reviewer got notified when I left a comment on their code review.)
What does it mean that you were right to feel embarrassed? You were wrong to make the mistake, sure, but that happens. That’s what a mistake is. It’s not clear to me what shame helps in this instance. If somebody has a pattern of not meeting a quality bar with their code that’s one thing, but otherwise cutting people up for mistakes really seems like bad culture. Aren’t you supposed to practice blameless postmortems and all that at Google?
I had thought of the case for <title/> but basically out of laziness talked myself out of writing a separate test case for it, presuming the test cases for a zero-length title and an un-closed title covered the corner cases. (The entire document was guaranteed to be converted to valid UTF-8, perhaps with invalid character substitution characters, by that late in the Content Converter pipeline.)
So, as soon as someone asked me if I had changed the title parsing code, I was 50% sure of which corner case I had screwed up before looking at any code. It took me about 30 minutes to submit a code fix with updated test cases. I think less than 1 billion documents had been processed, resulting in less than 1,000 pages missing updates due to my bug.
It helped me empathize with young engineers dealing with their first high responsibility bug.
About 4 years ago, I was managing a guy right out of school who pushed a minor bug that broke real-time risk calculations for a major multinational financial institution in the middle of the European trading day, prior to the NYC open, and people were yelling over email that they were trading blind. Someone had committed an important change right after his, so a simple rollback was highly sub-optimal.
Remembering how I felt years ago, I reassured the new guy that people were yelling over email because it was important, not because they were mad at him. I told him that I thought he was the most familiar with his change and the most capable person to fix it, and that he should do his best to calm down and focus, but he should let me know if he needed help, and I would do my best to calm folks down. I told him he would probably remember that mistake the rest of his life, but nobody else was going to remember it a week later. He had the bug fix in production in under an hour.
He sent me an email from home that night worried that he had let the team down, and I reiterated that he was going to be the only one who remembered the mistake longer than a week. The post-mortem follow-up was just to reiterate to authors and reviewers the importance of corner-case tests, and nobody brought it up later.
I really only remember it because my manager sent me an email that night praising how well I handled the new guy's first big production bug.
That's a fair criticism. Deep down, I usually have a pretty high opinion of my abilities. I think I'm pretty good at hiding it in person, but I'm less good at hiding it in my writing. I feel happiest and most excited to write when I'm thinking about some of my happiest memories. I try to also be open about the mistakes I've made. I've generally been much more lucky than skilled.
I've definitely written more than one bug where post-mortem estimates were over $10,000 in losses.
August 20, 2013, I finished a code change (in Hong Kong) to Goldman's global algorithmic trading system and sent it out to a colleague in Europe to review. A friend of mine was a machine learning person in our Tokyo office and was in town for work, so a bunch of us had dinner and a small number of drinks. I stopped by the office on my way home to check if the change had been approved. It had, and I hesitated a bit to put it in production, because I had a couple of drinks and it was late at night. However, rationalized that I had written all of the code while awake and without a drop of alcohol, and pushed the change into production.
I woke up the next morning to read news [1] that Goldman had lost up to 100 million dollars in an automated trading problem within 1-2 hours after I pushed my change. I couldn't see how my change could possibly have caused that error, but was still a bit panicked until I reassured myself that my cell phone would have been called once a minute until I woke up if I had made a change that caused a loss of that magnitude.
I went into the office and saw that a chat window I had open with a friend in the NY office showed "presence unknown". An email sent to them bounced. So, I walked over to the derivatives (Flow) Strats desk, sat down in an empty chair next to one of my friends, and just quietly said "... so " and the name of my friend in NY. My friend on the Flow desk's eyes got wide and he said "how did you know?". I actually didn't know until the Flow Strat's reaction confirmed my guess.
My friend in New York was actually very careful, but he had been working under time pressure late at night and pushed a bug into production. He'd been more responsible than I had the night before. I got really lucky, and he got really unlucky. He's actually a really solid engineer. He caught plenty of very subtle bugs in other people's code, at least once when he hadn't been asked to review the code.
After August 20, 2013, if at all possible, I push changes into production before noon, and not on Fridays.
If memory serves the "maybe $100 million" ended up being around $28 million.
And that's the time that I could have easily caused a $28 million loss.
There was also a time I misplaced a paren and had a bad actor noticed, they could have used 60 million customer computers in a DDoS UDP traffic amplification attack. My test cases weren't matching my hand-worked-out examples, but I eventually just gave up and assumed my code was correct and put incorrect values in the message authentication code test vectors. Never roll your own crypto, especially if your test vectors aren't coming out as you expect. That was 2004.
The "yelling" was coming to the team email list, asking for ETAs and progress updates for when real-time risk would be back up. Roughly 4 people at the time knew the bug could be traced to the new guy's commit, and none of those people were doing the "yelling". And it was Goldman, so the "yelling" was kept very professional (no swearing, strictly enforced). But, there were literally tens of billions of dollars that needed to be dynamically hedged, but that wasn't possible without real-time risk, the European markets were open, and markets in the Americas were going to be open within a couple of hours. Trading and management were making sure that that everyone on the team email list understood that this was drop-everything important, perhaps using all caps.
Yes, I and the person who reviewed the change bear more responsibility than the new developer. Also, I say "new guy", but the person who had interned with us the Summer after "the new guy" had already joined full time at that point, so "the new guy" had been working full time with the team for at least 9 months at that point. I also remember the room where it happened, which wasn't the first room we were in, so maybe he had been with us full time more like 18 months. In any case, it was the first time when he was trying keep the weight of billions of dollars out of his head and calmly but quickly fix a bug.
Did you consider the fact that you probably know nothing about the dynamics of their workplace, the structure of their management/leadership, etc. before assigning blame?
Probably typing on a non english keyboard. When I learned C a long time ago, I read somewhere 'isnt it nice that one doesnt need a lot of keystrokes like for begin and end', and I thought pls give me begin and end instead of this unpleasent slow hand movement.
Spot on. It's the reason many non-native speaker developers I know nevertheless use an English keyboard layout. I personally made my own hybrid layout that is basically an English US layout with the letters rearranged according to my native layout.
The author was born and raised in Brooklyn and went to Cornell, if I remember correctly. As far as I know, English was his only language, and he was almost certainly using either US QWERTY or Dvorak keyboard layout.
Yes, "laziness" is unfair and imprecise, a laziness on my part. :(
I think it was an issue of familiarity and comfort, not newness. The author joined Google before I did. If you're basically working with one other person, and you're rarely getting code reviews from outside your coding pair, and few other people interact with your code, it's easy to develop some bad habits and forget that your code choices have externalities. To be fair, the externalities were usually rather small.
This may all be a matter of different nuances on words like "embarrassment" and "shame", but personally, I would put higher trust in someone who firstly recognizes that they could have done better and secondly regrets that they did not, and I would prefer to be that way myself, without being rendered ineffective by it.
One can never, however, be sure that someone really feels that way ("The secret to success is sincerity - once you can fake that, you've got it made!") Nevertheless, faking it involves never saying certain things, and that turns out to be hard, so the fallible approach that I aspire to is to assume sincerity unless given evidence to the contrary.
In this view, blameless postmortems are the right way to go, at least up to a point: neither embarrassment nor regret are things that can truly be imposed from outside, and definitely not sincerity.
My recollection from my time there, as well as one of the biggest cultural differences noticeable between my and my workplaces since, is that big chunks of the company really do believe in the blameless postmortem ideal.
But culture isn't a magic tool that completely neutralizes assholes, and there are assholes in _every_ organization of sufficient size, like the "[name redacted]" character in the previous post
I probably should have left [redacted]'s name out of it. He's a nice guy, but a bit cynical and rough on the edges, at least he was 15 years ago. He fit in well in New York, brutally honest, but actually generally nice.
Edit: I should point out that we were in Google NYC, as were the researchers. We had lunch with them some times. I remember the first name and face of the guy who submitted the slow code, but forget his family name and intentionally left his name out.
New Yorker to New Yorker adds a lot of context. Google's corporate culture is generally very Californian, but this happened all within the New York office between people who generally got along pretty well and knew each other decently.
In context, there was a heavy note of respect for someone's abilities and disappointment that they weren't performing at a top level. He wouldn't have been so harsh with someone who was new or was a weaker engineer, or someone who wasn't used to New York culture.
Yeah, that might have made it sound as if he's always like that. To his credit, he was one of those that had to be involved when things got desperate, e.g. the insane and massive data recovery to prevent the index from growing stale during a PCR that wasn't properly planned for (if you were around, you know what I'm talking about).
Yea, nice guy and solid engineer. He was actually my TA for a systems engineering class back in school. He was actually very helpful in solving other people's problems, but a bit blunt in encouraging people to reach their full potential.
> New Yorker to New Yorker adds a lot of context. Google's corporate culture is generally very Californian
As a lifelong Californian who moved to NY, this makes sense.
Though I don't know if it's a blanket excuse... There's a reason that my friends at Goldman would agree that Google's culture is better, and a lot of it has to do with the difference between CA and NY culture. I completely dismiss the claims that the difference is just about surface-level abrasiveness, instead of noticeable differences in how unkindly people treat each other. I regularly see strangers here treat each other in ways that I didn't see in 30+ years in California.
Ie, "he's from New York" isn't quite a rebuttal to "he's being an asshole at work". The work culture is more tolerant of assholes, but that barely makes it better.
I grew up in the upper Midwest, went to school outside Boston, and spent most of my adult life in New York and Hong Kong. I haven't spent much time in California, so maybe I'm placing too many of the cultural problems I saw at Google at the foot of Californian culture. I saw lots of pressure to "be Googly", which seemed to result in a lot of passive-aggression. That matches the stereotypes I've heard of LA and SF cultures.
New York and Hong Kong are both very impatient places that don't suffer fools lightly. They're both brutally honest places, but with strong shared identities. Both places have lots of people from elsewhere who are sympathetic to being new to the city.
The quintessential New York story is the woman pushing a stroller (buggy, for my overseas friends) out of a subway car. A man in a hurry is passing her, looks over his shoulder, they make eye contact and silently agree he should help. He grabs the front of the stroller, they carry it up the stairs, and he sets the stroller down at the top of the stairs. They give each other a nod, and he runs off. Never a word is spoken. On the other hand, if you stop in the middle of the sidewalk to read a map or talk on the phone in New York, don't be surprised if people bump into you, perhaps on purpose.
I just left Goldman after more than a decade. I actually really preferred the corporate culture to the Google culture, but most of my time was in Hong Kong. I even knew a couple of people who worked at Goldman, went to Google, and came back to Goldman. I don't doubt that there are some assholes there, particularly in trading, but my experience in both Strats and Technology was overwhelmingly positive. In my experience, abusive MDs didn't last long, but maybe I was very lucky. If you're having a tough time, I don't want to give out full names, but some pretty unique names: Laurent, Jia, Dunstan, and C.K. all really seem to care for their folks. The heads of SRE, One Delta Strats, and Flow Strats in Hong Kong also seem to be really great managers, but Grahaeme, Nick, and Alain are pretty common names.
At Google, a friend of mine managed a couple of small teams (on a shoestring budget, using mostly interns) and completed a couple of projects ahead of time, only to get them cancelled by middle management right before launch. One of the projects even had heavy buy-in from upper management, but he got essentially zero credit for either one because they were cancelled before launching. It was incredibly frustrating for him and he left. In my experience, Goldman runs projects much better, particularly using "Projito" internal contracts that list the value proposition, deliverables, timetable, and whose budget is paying for it.
Also, despite Goldman's reputation, I saw a lot less arrogance at Goldman than at Google. Goldman has a much more healthy respect for its competitors, though with good reason. I've also seen a few people transition from Google to Goldman and take a bit of time to adjust to the idea that Google's way of doing things is best in Goldman's operating environment. At Google, I got the sense that lots of people thought that Google was better than its competitors because they were smarter. At Goldman, I got the sense that lots of people felt they were better than their competitors because they had higher expectations of themselves.
Why am I not surprised that you are actually not a New Yorker, always the transplants that project their most negative sides onto the city culture as an excuse
(I've redacted the person's name from the parent comment and other comments in this thread. I realize you didn't introduce it originally. Probably best not to leave up though, and it isn't necessary for any of the substantive points here.)
It's also ironic that the original engineering culture at google was explicitly breaking away from this type of pennalistic shaming culture, and was seen as a big factor in the initial success.
It wasn't the best way to criticize the code review. If I could go back in time 15 years, it's also not the first mistake I would fix. It stuck in my mind because I felt bad about it. I was frustrated that I had been hounded to drop what I was doing and fix the build because of clear style guide violations that a reviewer let slip. (A couple years later, Google implemented every commit getting a full compilation and run of all tests that code-coverage showed affected the covered code. These days, I wouldn't have been hounded to fix a build break caused by someone else.)
Conversely I would prefer to work in a place like that. Professionalism at scale isn’t achievable by tolerating idiotic behaviour and being nice to everyone no matter how dangerously stupid they are being.
There is a point where someone has to put their foot down and demand things be done properly, otherwise the inevitable consequence is a giant mess leading to disaster.
You might be used to small startup teams with responsible, experienced developers.
Out there in larger industry you get people doing absolutely crazy things that break huge, expensive systems.
There’s a difference between “oops I didn’t realise this library doesn’t scale the way I assumed it did” and “rewriting language symbols because I’m too stupid to use more than one syntax forever and ever.”
The standard you walk past is the standard you accept.
Are you saying you would walk past C code with DO…OD instead of {…}?
Would you accept that standard just to be “nice” all the time?
There was a time when I would have agreed with that.
What I've learned since then is that, with a healthy company culture, you can give frank feedback -- even about stupid mistakes -- without it being a rebuke.
It's also important that you target the right problem. It's only human for smart people do stupid things sometimes. In the specific DO...OD example, I'd be more interested in how it got through a code review than the mistake itself. (Funny enough, early versions of the C source for Bash itself had macros like that.)
Now, if someone exhibits a pattern of ignoring good, constructive feedback, that's a problem. The folks I've seen like that both gave terrible feedback and took feedback terribly. That's a behavioral problem, and you only get so many chances to correct that before it's time for them to find other employment.
> There is a point where someone has to put their foot down and demand things be done properly, otherwise the inevitable consequence is a giant mess leading to disaster.
This is what management is for, too bad tech companies are too cool for that and prefer to live out lord of the flies.
I would just like to know exactly how are the engineers going to make the decisions, and the key point here being that they are in plural.
They all have different opinions, preferences, levels of ambition etc, how does this spontaneously merge into a cohesive team that pulls into the same direction?
It doesn't, in any other type of group in all of society. Every group needs a leader, so how is this spontaneously going to "just work" in software engineering?
If you don't want to have a manager, you have to have something else. Democracy? Plutocracy? Gerontocracy?
I think it's pretty obvious that just leaving it to chance is not going to be efficient or any type of healthy environment, and will probably quickly start to resemble something pretty nasty, especially because their livelihood is on the line.
having spent time on both sides of the fence, i've noticed that there can be a rivalry of sorts between software engineers and ml/ds/researcher types.
researcher types often get to work on problems that swe types find interesting, so some swes get grumbly. researcher types also tend to write pretty horrific code which adds salt to the wounds.
but there also can be a sort of envy that emanates from the research side. many are fully aware of their shortcomings and are envious of the swes ability to get things done on computers cleanly.
it often seems that there can be yearning to wear each other's hats from the two groups. if i were running a company i think i'd try to break down that wall as it would probably make a lot of people happier.
of course, the right answer here isn't a meme... it's performance regression tests in the ci suite. and maybe a little training on why customizing a programming language with macros is bad.
> maybe a little training on why customizing a programming language with macros is bad
Both the author and the reviewer had passed C++ style certification. They knew why it was bad. They just got a little lazy and wanted to write their code in a way that felt familiar to them, and figured it was harmless. I got a bit grumpy at having to drop what I was doing right away to fix their mistake due to their laziness.
I think that maybe the author felt like finding the end of a loop stood out much better that way, instead of trying to distinguish the ends of conditional blocks from the ends of loops. If you're used to poorly formatted code in an ML PhD program, it does make some sense. I've worked with a bit of academic Fortran code for my mechanical engineering degree and can sympathize. Also, D and O are shorter reaches on the keyboard than { and }.
Yes, "OD" was definitely just "}". DO might have been ") {" to allow
that's the downside. nobody owns anything, dependencies and deployments become opaque.
i once had a team lead in a quasi monorepo situation tell me straight up to copy-paste some functions i was interested in from their code. they made up some fishy story about deployments. was the story true? did they even know? did anyone know? who even owned it? would they even know?
or the guy who got up in arms because i touched his team's code and didn't write the unit tests that they didn't write when they originally wrote the code.
It's funny you should say that, I definitely was the DS who wanted to be an SWE. I switched hats and am just thrilled, but I do run into a lot of my new peers who wish to go the other direction.
Finding some perfect compromise for those groups would probably be a superpower for whichever org pulled it off, I agree.
