> Despite the unreasonable effectiveness of simple architectures, most press goes to complex architectures
We're over-simplifying these terms. What's simple in one way is complex in another way. It makes for great blog posts to use generic terms like "simple" and "complex" for broad-ranging and highly technical topics, but it means nothing. Be specific or you're just tilting at windmills.
> The cost of our engineering team completely dominates the cost of the systems we operate.
Could this be because you decided to invent a new kind of truck, thus requiring lots of costly engineers, when getting two mechanics to weld a crane onto a flatbed would have done the trick?
NIH is rampant in tech. Everybody thinks they need to invent something new when they can't find off-the-shelf parts that do what they want. When in reality, just making their system a bit more complicated in order to accommodate off-the-shelf would have been faster and cheaper. Sometimes more complex is simpler.
Later on in the article Dan mentions build vs buy, but it's not that simple. You can buy and then modify, and you can build in a way that's cheaper and easier, if uglier and more complex. Design and engineering decisions aren't binary. There are more options than just "only use a complete product" vs "build your own everything", and more business and engineering considerations than just time to market and "scale".
> Rather than take on the complexity of making our monolith async we farm out long-running tasks (that we don’t want responses to block on) to a queue.
See, here's the thing with "simple" vs "complex". It's not "simpler" to use a queue than async. Both are complex. Queues may look simple, but they're not. Async may look complex, and it is. You didn't pick simple over complex, you picked something that looked easy over something that looked hard. You made a bet. Which is fine! But let's not pretend this is somehow good design. You just ignored all the complexity you're going to run into later when your queue becomes a problem. Maybe it won't be a problem? But maybe async wouldn't have been a problem at scale either. "Maybe" doesn't equal "good design", so let's not pretend it does (re: the idea that it's simpler and thus better).
> Some choices that we’re unsure about [..] were using RabbitMQ [..], using Celery [..], using SQLAlchemy [..], and using Python [..]. [...] if we were starting a similar codebase from scratch today we’d think hard about whether they were the right choice
This is a sign of a lack of engineering experience. I don't mean that as a diss; I only mean that the correct choice (of these options) is clear to somebody really familiar with all these options. If you don't have that experience, you need to go find someone that does and ask them. This is one of the best reasons why you don't want to engineer something yourself: you probably aren't the best engineer in the world, and so probably don't know the kind of problems you are going to run into. However, if you are engineering something yourself, and don't know the right answer, you need to go ask somebody (or many somebodies). If you were inventing some new kind of device that nobody had ever invented, maybe nobody knows? But you probably aren't doing that, so somebody does know. And if for some reason you can't or won't go ask somebody, then you should use the thing that you[r team] is most familiar with and already know all the pitfalls of, regardless of nearly any other downside from its use. Unexpected problems are a much bigger thing than expected problems.
> By keeping our application architecture as simple as possible, we can spend our complexity (and headcount) budget in places where there’s complexity that it benefits our business to take on.
And this is still good advice. But, again there is no such thing as a binary choice of "simple vs complex". It's all shades of gray.
> Despite the unreasonable effectiveness of simple architectures, most press goes to complex architectures
We're over-simplifying these terms. What's simple in one way is complex in another way. It makes for great blog posts to use generic terms like "simple" and "complex" for broad-ranging and highly technical topics, but it means nothing. Be specific or you're just tilting at windmills.
> The cost of our engineering team completely dominates the cost of the systems we operate.
Could this be because you decided to invent a new kind of truck, thus requiring lots of costly engineers, when getting two mechanics to weld a crane onto a flatbed would have done the trick?
NIH is rampant in tech. Everybody thinks they need to invent something new when they can't find off-the-shelf parts that do what they want. When in reality, just making their system a bit more complicated in order to accommodate off-the-shelf would have been faster and cheaper. Sometimes more complex is simpler.
Later on in the article Dan mentions build vs buy, but it's not that simple. You can buy and then modify, and you can build in a way that's cheaper and easier, if uglier and more complex. Design and engineering decisions aren't binary. There are more options than just "only use a complete product" vs "build your own everything", and more business and engineering considerations than just time to market and "scale".
> Rather than take on the complexity of making our monolith async we farm out long-running tasks (that we don’t want responses to block on) to a queue.
See, here's the thing with "simple" vs "complex". It's not "simpler" to use a queue than async. Both are complex. Queues may look simple, but they're not. Async may look complex, and it is. You didn't pick simple over complex, you picked something that looked easy over something that looked hard. You made a bet. Which is fine! But let's not pretend this is somehow good design. You just ignored all the complexity you're going to run into later when your queue becomes a problem. Maybe it won't be a problem? But maybe async wouldn't have been a problem at scale either. "Maybe" doesn't equal "good design", so let's not pretend it does (re: the idea that it's simpler and thus better).
> Some choices that we’re unsure about [..] were using RabbitMQ [..], using Celery [..], using SQLAlchemy [..], and using Python [..]. [...] if we were starting a similar codebase from scratch today we’d think hard about whether they were the right choice
This is a sign of a lack of engineering experience. I don't mean that as a diss; I only mean that the correct choice (of these options) is clear to somebody really familiar with all these options. If you don't have that experience, you need to go find someone that does and ask them. This is one of the best reasons why you don't want to engineer something yourself: you probably aren't the best engineer in the world, and so probably don't know the kind of problems you are going to run into. However, if you are engineering something yourself, and don't know the right answer, you need to go ask somebody (or many somebodies). If you were inventing some new kind of device that nobody had ever invented, maybe nobody knows? But you probably aren't doing that, so somebody does know. And if for some reason you can't or won't go ask somebody, then you should use the thing that you[r team] is most familiar with and already know all the pitfalls of, regardless of nearly any other downside from its use. Unexpected problems are a much bigger thing than expected problems.
> By keeping our application architecture as simple as possible, we can spend our complexity (and headcount) budget in places where there’s complexity that it benefits our business to take on.
And this is still good advice. But, again there is no such thing as a binary choice of "simple vs complex". It's all shades of gray.