The thing that makes it difficult to imagine is that the large numbers involved are hard to imagine.

In a given teaspoon of ocean water in the top layer, there are millions of bacteria (in soil there can be up to 1 billion). Each one lives for a day or two before it either divides or is consumed, with a handful of mutations at each round of division. So ~200 divisions a year, for three billion years, with selection stochastically whittling out the few good mutations the crop up now and then, in a diversity of changing ecosystems and you end up with where we are today. Oh, and the occasional horizontal gene transfer for extra spice.

Obviously that is a large hand wave -- the numbers above are from today's environment; early on the biotic density was lower. But the large numbers swamp things. The only real mystery is how things initially got started. But again, it is hard to imagine the time scale involved and the wide variety of environments that exist over the time to imagine the happy accident where the first self-replicating molecule just happen in the right environment that was stable enough for long enough for that self replication to gain traction.

Have you actually seen the results of things made with evolutionary algorithms? They ALWAYS produce "what the fuck, how could this be good" outcomes. From antenna that look like scifi props, to a computational system that somehow requires a supposedly unconnected transistor to be activated, evolutionary systems always find their way to a goal in an unimaginable fashion, because "random" mutations are basically the direct opposite of engineering something, so why does everyone always expect the outcome to look engineered?

Evolution producing a complicated, half non-working, incomprehensible, "everything interacts with everything else in a chaotic and unpredictable to us way", is the EXPECTED outcome.

It's similar to how many big programming projects become spaghetti messes of half integrations and barely functional parts hooked together half-hazardly where every feature relies on a bit of code nobody understands. It's an "iterate on the stuff that works" process, except the machinery inside a cell is way more effective and tolerant of such a regime than our stupidly strict programming languages and computers.

Well, that is true. But presumably selection pressure —- namely, the need for things to work reliably and in an organized fashion —- imposes structure on systems that parallels the kind of organization that engineers use to make complicated systems work reliably. There’s a reason that complex organisms concentrate specific functions into organs with recognizable interfaces rather than scattering those cells widely throughout the body, in the same way that a human-engineered mechanism is usually constructed from parts. The fact that this allows for organ transplants isn’t really by design, but it’s a convenient outcome.

It's still incredible despite all that. Not in the sense that one chooses not to believe it, but in the sense that it's hard to fathom that it actually happened.

First you need the incredible machinery to enable mutations in the first place

I'm curious: if tomorrow someone discovers that some parts of the genome are encoded with RSA cryptography, will the argument of "mutations, time and selection pressure" still hold? The same handwaving applies :-)

It 100% still holds.

The convergent evolution of pitcher plants is an example of this. There are a number of features that evolve to the same functionality in plants, and many of them have to work together to become fully functional. Yet we see in plants separated by vast distances and millions of years of separation that traits that are useless alone encode themselves and then will will have near spontaneous usefulness when some other gene evolves.

The laws of large numbers are not things the human mind really grasps well at all.

So if a monkeys bangs on keys forever it will eventually produce Shakespeare? I'm quite skeptical

There's no selection pressure when a monkey bangs their hands on a keyboard. It's just randomness without any consequence. The letters are not interacting with each other or their environment in any significant way.

What about offering the monkey some peanuts if he bangs correct words, the more correct words the more peanuts he gets?

I understand there would be limitations in the lifespan of the monkey and such, but in a hypothetical case, would that work?

Evolution wasn't evolving towards the specific point we are at now. It was just doing its own thing all the time, with no plans for the future.

So, a better (but still not great) question, would be, if a monkey bangs on keys forever would they ever produce any kind of story at all?

But, that's still not good enough, because evolution is iterative, and would pick out very short stories that worked a bit, and that page kept getting duplicated out to new monkeys who would start typing from the end of that story

Obviously yes, hence we are the monkeys banging on the keyboard. Evolution just boosted those that made interesting strings because it gave a survival advantage.