This might not help you, but it might help someone earlier in their career.

I was born, raised and studied in South Africa. Living costs and salaries are cheaper. I worked in California. Instead of living a lavish lifestyle, I saved. In hindsight, I should have saved even more. The important thing I did was opting for cheaper housing. I worked hard, which opened doors. I landed up at a startup that's now doing really well.

I retired after just six years. Six more years later, I'm doing what I want. I've been cycling around Europe and Africa. Next week I'm flying to Canada to explore North America for the next two years. It's a pretty cheap lifestyle, but I get to experience life around the world in a way few people ever do. I'm working on building a presence on YouTube. I've met others who sustain their travels via YouTube. Even if I don't, I can keep going for quite some time living off of savings. I wouldn't be able to do this so easily if it weren't for stock.

I'm not advocating a travel lifestyle. Instead, I'm advocating for saving up while you're earning decent cash. Don't blow it all. Then hopefully you can leave for what you really want to do, and not be tied down due to finances.

Not only is this unrelated to the OP, but it's also not actionable advice. Sounds like you got lucky and got a bunch of great stock. That's not something most people can count on, and housing in American urban centers only goes so low.

"Save while you're earning decent cash" seems pretty actionable to me.

I found the anecdote of retiring in 6 years to distort the message significantly. Sure everyone can save more but that likely won't result in massive lifestyle changes in a short time frame.

Kind of depends. Plenty of people COULD live like a monk and save a ton over 6 years. But that requires not having a family and likely not enjoying your life very much. I probably COULD live on significantly less than I make right now. Hell, I have in the past.

But I sure didnt enjoy it very much.

But if you lived on rice and beans in the cheapest place you could find while making a 6 figure salary... Saving a very significant sum every year would be possible.

Six years is certainly lucky, but it's easy to save working as SWE. There are people living in your city on a tiny fraction of your income. A person without commitments could save the entire diff, if they wanted to.

And travelling is cheap as well, I have a friend who is travelling long term for like $300-600 a month, including all the transportation costs, etc. (The opportunity cost of not working is obviously much higher though.)

Yeah for real. I think that any dev can retire by 50 if they want to. Maybe by 40 if they really put in effort, but if you're retiring before your mid 30's you're either 'leanFire' meaning you've basically committed yourself to a life of relative poverty or you somehow got hold of a 'lotto ticket', and that, by definition is extremely rare and not something anyone should plan for.

"I wouldn't be able to do this so easily if it weren't for stock" does not. It's misleading to attribute early retirement to a frugal lifestyle and savings when it appears that they hit the lottery.

> "Save while you're earning decent cash" seems pretty actionable to me.

This is in fact pretty unactionable unless you are planning on boondocking during your career and retirement - at which point - why bother with post secondary education and a white collar job in the first place, just boondock from the getgo.

I was able to save around ~$800k over 6 years while living in the Bay Area, that's now at $1.45mil two years later. But I consider myself extremely, extremely lucky.

It's incredibly unrealistic for anyone to replicate that sort of principal savings and then return. You might as well suggest buying lottery tickets, which you essenntially admit. This is either bad advice or shameless bragging, but tough to see it in any charitable light.

OP asked for any kinds of stories.

Do you mind sharing your total comp during those years? (since you are sharing pretty personal numbers already)

TC was ~300k USD. My base salary could cover my living expenses (single living with a roommate). I held all of my RSUs + ESPP for 5 years.

How old are you? Do you have any dependents, a partner, children? Do you have any health issues; what do you do for coverage? What are you going to do in 'n' years when you're older, less relevant and broke?

I too am a saver, and over the past 20+ years we've been suckers. The problem is I grew up as a child when a savings account, essentially a zero-risk asset, paid well over 5% interest. That hasn't been true for a long time, with inflation & taxes grinding away at any marginal gains. I wish I'd financed and leveraged more, not consumption goods but investments. The other problem with being a saver is it's very hard to be one, then flip and be a consumer of your savings. The thing that makes you able to save is also what holds you back from spending. There are a lot of baby boomers who have a lot of assets yet still live very thrifty lifestyles; especially if they grew up blue collar when you could "save yourself rich".

>I wish I'd financed and leveraged more, not consumption goods but investments.

The good news is, this is available to you right now!

The bad news is buying the best assets is hard, just like it was 5, 10, and 20 years ago.

Savings accounts weren't paying 5% for nothing. It's easy to look back at what could have been, but those moments are happening right now, too.

So long as you are single and childless.

fair point [0]

[0] https://www.pewresearch.org/short-reads/2023/02/08/for-valen....

As an early career I agree with almost everything you say except doing it to quit and not work in industry

I mean, I save quite a bit (more than a third after taxes) just because consumerism is annoying. But at the same time getting paid to write code and learn how computers work seems like a pretty sweet gig to me long term. Especially because there's a compounding effect, the more you know the easier it gets to learn new things

And big company bullshit isn't that bad, at least to me. It's a bunch of convenient things to complain about

OK now let's compound this and think about how society would look if every participant followed the "rip and dip" methodology.

The actual advice you're giving here is generally good, but for your specific outcomes there's a relevant XKCD [0].

[0] https://xkcd.com/1827/

You want to replicate as much as possible, but if we ran canary on the same machines we could have testing code bring down production. That's bad.

We actually have a traffic replica (dark client) setup for the new webserver architecture we are gradually migrating to. It likely would have caught this before deploying to users.

We have very comprehensive dashboards. Getting the perfect ones that help in all cases, while not being information overload (the problem here) and being discoverable is a hard, iterative process.

Yes, monitoring requires a lot of tuning until you find a sweet spot, but it doesn't sound like this is something that would've been buried deep in the annals of monitor. CPU/load data on your web servers should be pretty visible/accessible and one of the first graphs that get pulled up (and your alarms should've pointed out the issue anyway).

I'm not sure what you're using for dashboards but Datadog makes it pretty easy to find this stuff. I'm not a Datadog shill and I actually am not a huge fan of the product, but it's what we use and it's been a big help over our previous Munin installation.

Other process changes that could prevent this are good load testing in a stage environment and getting your company using the real prod code on the real prod infrastructure as its main/default install. A lot of the benefits of "dogfooding" are lost if you're using alpha code on dev-only boxes (as you state that you are in another comment).

As another commenter said, I'm not sure that postmortems like this are valuable unless the problem was particularly complex/interesting. I'm sure that a lot of people at Asana know how to fix this and that it's just a matter of getting management to allow them to do so. I'm sure you owe your customers an explanation of some sort, but I don't know if you need to get into details that say "Yeah, it was just a pretty typical organizational failure, we really should've known better". Everyone has those, but it's best not to publicize them too much.

I'm not going to hold it against Asana because I've worked at a lot of companies and I know how this goes, but when people come here and analyze the cause, as a postmortem invites the readers to do, you seem a little defensive. Perhaps it's best to keep the explanation more brief/vague when it's not a complex failure.

When you do daily deployments, you can't QA every one much. You rely on automated tests and Internal users using the new code for a couple hours before the deployment. We were unlucky in this case with the number of bad releases. Each was relatively minor, and ironically one was to fix a bug with the code that caused this outage. We run a 5 whys for most of them.

> When you do daily deployments, you can't QA every one much.

In that case, should you be doing daily deployments to production?

I include automated testing in my definition of QA. (Necessary but not sufficient)

Are the daily drops predominately bug fixes or also a regular drip of new functionality?

I think the old world of quarterly releases was also bad for other reasons. I'm curious about the right middle point.

Every time a company like Asana comes clean about outages and software quality issues, the canon of knowledge improves. Thank you for sharing!

We will never out individuals. The person who committed the code was innocent. We got him a fun gift as a sort of joke.

Yep the best thing some could do:

- train the people more - help them to get over (some ppl could be really mad and infconfident after they did bad)

That was entirely tongue-in-cheek. I wouldn't ever expect you to do that! It was an exaggerated example.

Every response from our users so far has been thanking is for the transparency. It also represents our internal transparency, and that has a real impact on recruiting.

Not an Asana user but if I were this kind of response is exactly what I like to see as a both a user and a developer so well done.

We do, but it didn't help given the cause of the high cpu was our logging infrastructure (Amazon Kinesis) being overloaded by the webservers.

Does kinesis not support UDP sylog style logging, some of these old technologies had the right idea: if your sending too much data, drop the packets on the floor instead of falling over!

This was the first time we had this class of outage. Many things were in a very bad state, and many of these symptoms were more familiar to us. So we spent time ruling them out before realising webserver CPU was closer to the root cause than the other symptoms.

We roll back by reverting to a previous release on the load balancers, which is usually pretty instant. The previous releases were bad and themselves rolled back, which is a rare situation for us. So there was a bit of scrambling to look into the chat logs to determine a safe (non-rolled back) release we could roll back to. Then the high CPU caused our roll back to be really, really slow. Then we still had old processes running the bad release running, and killing them on webservers with high CPU took a while to actually work. Then it took a bit of time for load to come down on its own. All of this took place within the 8:08-8:29 window reported in the post. And I'm still simplifying a lot.

What I don't get is why you didn't see the relatively low cpu usage on the database server and the super high ones on the webserver immediately in a nagios (or similar) dashboard.

They were distracted by the previous experience of having issues elsewhere.

And apparently there were no alarms in place for these kind of things

Apparently a lot of parts of the system were on alarm.

It's because they don't have a simple rollup dashboard that you can see that at a glance, like most places. Can you imagine if your car just showed you an event log for a door open, oil, turn singles on etc. that's what most monitoring systems are like these days.

Roll backs are in chat logs? I'd assume your scripts would record what they do when they do it, including roll backs.

Also, when only deploying two times a day, it's harder to tell which of the included changes have the problem. That's an argument for more frequent deploys!

Seems like pretty ambitious logging that it tripped the servers !!! Will be careful with my logging next time :) .

Out of curiosity, why are you deploying to all your web servers simultaneously? Could you not do a partial roll-out to make sure something like this doesnt happen?

I doubt partial roll out would have helped in this particular case since it only happens in high load and they roll out new code twice a day.

Correct. We don't roll out during peak load either.

Considered at least starting your release canary during peak load?

We have talked about it. It is unlikely to helped with an event like this, and I don't recall an event where it would have. It also has the downside of extending our deployment cycle by a lot. Notably, we do run a canary internally, and that had no issues, which actually through us off for a while because while the app was partially down for users it was working for us and that hasn't happened to us in a while.

I work at Asana. There's no way we'll drop our free plan. It's too fundamental to our sales model. We are, however, thinking a lot more actively about what features to make premium. Largely new features we haven't released yet.

I believe you're being honest, but what about in two years? Four years? Six years? Anything can change in that time span, including your sales model.

Not that this is a dig against you or Asana; it's just that this is how the world works.

While that's true, the implication of the OPs statement is that the free plan does a great job converting users to the paid plan. While that's true, they'd be crazy to ditch it.

More likely is that products come and go.

We're definitely not getting rid of the free plan. Well, not until current management leaves, we do badly, get greedy, get bought out...

Yeah, jerk! Answer for your unknowable future! "No way", indeed... the temerity, to suggest their product will be free after the heat death of the universe.

Your response seems like an unnecessary escalation. I thought the post you're responding to was reasonable.

I'm pretty sure it's a joke.

Jokes and other forms of amusement are banned on HN. It's been found that people here take things way too seriously, so the mods decided to ban jokes and humor outright.

(Happy April 1st!)

You know, I've had my share of rants, etc. on HN but revisiting Reddit has made me appreciate the HN policies more.

I generally avoid Reddit these days, except occasionally dipping into the VR sub-reddits. The level of stupidity, entitlement, and general melodrama is ludicrous.

It's like when you're subjected to the nonsense of a person who's become accustomed to being the smartest person in a dumb room.

HN is just a smarter room. Like Reddit 10 years ago. We should defend that.

Right. HN's lack of drama and brigading, and the fact that it's a small community, makes it a lot of fun.

(I just lament a slight lack of light-heartedness!)

> until current management leaves, we do badly, get greedy, get bought out

I think that's the point of the question?

Think he was making a joke (about how you never know what'll happen in future). Or at least I hope so.

I'd pay for Asana if I ever needed to manage a small team on a personal project or if I start my own company. There's just too much intertia for me to change the project management software at the clients I work with now as a consultant.