Hot code upgrades on the BEAM are awesome, but they're not a piece of cake. If you're also interested in the challenges of making them production safe, I gave a talk about this topic on CodeBEAM Sto earlier this year:
OP talks in the summary about the importance of understanding the process. It's very much true, but you need to understand not only the process your tooling provides, but also what's going on in the background and what hasn't been taken care for you by your tools. I'm afraid these things are rarely understood about hot upgrades, even by experienced Erlang engineers.
> arbitrary addition to Coati, "also known as....the Brazilian aardvark"
But via citogenesis, the coati really became also known as the Brazilian aardvark. So the original claim is true, and this wasn't really citogenesis after all. More like self fulfilling prophecy.
In linguistics a sentence that causes itself to be true is called performative. (The standard example being "we are at war", spoken by someone with the authority to declare war.)
That doesn't really fit here, but it's a similar idea.
Based on my quite limited experience with Python the big drawback of indentation sensitivity is copy pasting will only work when your editor has great language support. If I copy a block of code from indentation level X and paste it to indentation level Y when I have explicit end-of-block tokens the worst thing that could happen is that I end up with badly indented code that I have to clean up manually. On the other hand, if indentation matters, and my editor messes up things, I am very likely to get code that looks good, probably compiles too, but does nothing like the original. I have to read through every line carefully and think about where the blocks were meant to end.
In the end I'm still not convinced that significant indentation would worth the trouble.
I've been writing python for 20 years, and I truly cannot think of an instance where pasting code was an issue. I use vim mostly without any plugins, not any fancy editor.
I've only used python a few years professionally, but seen it multiple times. With other languages I paste in code between some brackets, run the autoformatter and it looks as I want. With python I paste in some code, run the autoformatter, only to see because of spacing issues half the lines I pasted didn't get properly parsed to be inside the for loop or whatever.
Also makes it hard to copy a snippet to run inside a REPL or whatever.
In general just a huge amount of pain, and the "good thing" about how it forces well formatted code I've never seen as an issue with autoformatters and linters in othet languages.
The "trick" is to copy and paste on the right block level. Just set the cursor appropriately (which the editor does for you if you use the feature to jump to block start or end).
Than everything just works.
(For that to work even better the stupid feature of some editors to remove white space form all empty lines should be disabled. So pressing enter or going to the next empty line will always position the cursor on the correct indentation level. Than it's easy to paste code blocks everywhere as even empty lines are correctly indented).
I've written a lot of python in my life (but most of the time not as my main day-to-day language) so whenever I had to try something small in a REPL I do paste. I guess it would work to highlight a piece of code and run it externally with a shortcut, but I've never set that up, so I did indeed run into this problem more often whenever I was just taking up Python again (for example I find [x for x in ....] kinda horrible and have at times fat-fingered if it's twice nested)
> If I copy a block of code from indentation level X and paste it to indentation level Y when I have explicit end-of-block tokens the worst thing that could happen is that I end up with badly indented code that I have to clean up manually.
That's exactly the same "clean up" you need to do if you did something wrong while copy & pasting code without the useless block delimiters.
The rest is made up as someone pointed out already: Exactly this story comes up every time but in reality it never happens. (Besides when someone tries hard to get it wrong by all means only to prove their made up point).
Good news is that today is Black Friday, so the e-commerce industry is running at peak capacity. In 30 days it will be Christmas, and by then (the very latest!) everybody will scale back, so you have a good chance to gain access to more compute before you reach the end of your runway.
I'm sorry to hear you got frustrated with this task, but to be honest, I don't think it would be representative of how it feels working with Elixir or Erlang. It's like sitting down with zero experience to an IBM mainframe and trying to accomplish some routine administration task, setting up mandatory password rotation policy or whatever. It'd feel frustrating and difficult for sure, but it doesn't tell mutch about the mainframe. Especially since it isn't even a design goal for it to feel similar to the Windows or Mac OS your familiar with.
It may or may not be fraud as the letter of the law, but this is definitely the shadiest part of the story. Also a pretty lame thing to do.
It would have been much more elegant to add some crazy overpriced option to your pizzas where you could make a huge margin. Like "gift wrap +$100" which would mean putting a $0.01 ribbon around the box. You could make much more money this way from Doordash with less hassle than switching certain orders to plain dough.
In the beginning (read: 2005) there was only one system, Kred. It was written in Erlang and did everything, so became a huge monolith over time.
About 10 years later Klarna started to change its system architecture (which makes total sense, since by that time it grew to a successful unicorn, with a way more complex business than what it had at launch time).
First, instead of shovelling every new feature into the monolith, it introduced new (mostly micro-)services around it. This model is successful, and most of these new services are written in Java, but there are also Erlang ones and other, even some "exotic" ones, like Haskell. (Also, through a series of acquisitions, Klarna got a bunch of tech from other companies, which, you can imagine, were written in whatever language those other companies used.)
There was also an attempt to replace Kred, the old Erlang system, with a new one (happened to be written in Java), and this project indeed was a failure in the sense that Kred is still around and nobody wants to decommission it nowadays. But the intended replacement system still proved useful, so it's not really a failure from that system's developers point of view.
Kafka being a soft dependency is not an assumption, it's a design goal. So the conclusion should be to eliminate any hard dependencies on Kafka, if anything.
However, when it comes to metrics in particular, I'd say Kafka was still a soft dependency. The reason we lost our metrics during the incident was a scheduler lock-up blocking the collection of VM-level metrics. It's just coincidence that the scheduler lock was caused by brod in this case. Otherwise metrics would just flow to Graphite directly, never interacting with Kafka.
When it comes to the complete monitoring solution around Kred, there was one bit of it depending on Kafka: System monitor (https://github.com/klarna-incubator/system_monitor). This tool is exporting data that doesn't fit well into Graphite or Splunk, so we store it in Postgres instead. Due to pure laziness it was at the time of the incident not writing directly to Postgres though, but was just pushing the data to Kafka. (The reason is that we already had a service available to push data from Kafka to Postgres.) After the incident we eliminated Kafka from the path. I didn't mention this work in the post because it was only marginally related.
I'd like to play with a game that's like Civilisation, but when you start, you don't know what world you're playing in. You only know what your ruler sees and hears. You may send your Columbus across the Atlantic, and you will see him arrive to India in the East. Later on you may learn that he instead discovered a new continent. Maybe. Maybe he really landed in India in your game. Or maybe he was a fraud and found nothing, and all the lands he reported on will disappear from the map when you send more ships to follow his route.
Similarly, you wouldn't know which technology would work in this world and which not. Maybe alchemy would be real, and you could develop it to mass produce rare materials. Maybe it would turn out to be fake science only. Similarly, magic and religion may work as either basic mind tricks and psychology that enlightenment would mostly cancel out, or be part of the reality of the world, and you could get gods fighting on your side Greek mythology style, or wizards casting spells even deadlier than tanks and nukes.
I guess this system would be already a bit too hard to implement, but if I could keep wishing freely, it would be awesome if you could actually govern by writing whatever law you want. So you wouldn't just click a button to switch from feudalism to theocracy or communism, but you would actually have to come to an agreement with power figures (or classes) in your society on how your state would work. You could grant rights to tax trade routes in exchange of doing military service for example. And later you would need support from some other group if you would like to abolish this system.
https://youtu.be/epORYuUKvZ0?si=gkVBgrX2VpBFQAk5
OP talks in the summary about the importance of understanding the process. It's very much true, but you need to understand not only the process your tooling provides, but also what's going on in the background and what hasn't been taken care for you by your tools. I'm afraid these things are rarely understood about hot upgrades, even by experienced Erlang engineers.