The ability of humans to get used to progress is amazing to me. We have a computer system that you can describe, in conversational English, the requirements for a program, including style, conventions and programming language, and this system will then synthesize this program from thin air, even if it's a novel idea with novel requirements, it will autonomously test it, make improvements and take your feedback into account. It will do this using a vast (if imperfect) knowledge of programming frameworks, far outstripping in breadth any individual human programmer.

And knowing this, you think that the only reason we could have to expect to create intelligence in a machine, even surpassing a human... is "Asimov-inspired, stimulant-fueled philosophizing"? That seems deeply unserious to me.

I find LLM's just as awe inspiring as you. But you and I both will find them underwhelming eventually. This is the history of the field going all the way back to Turing.

At any rate, my point remains. The flaws inherent to the current deep learning regime _absolutely_ disqualify them as being capable of any sort of rapid takeoff/escalation (a la paperclip optimization) that the rationalist community is likely referring to when they say super intelligence or ASI.

Sorry about the "asimovian" comment - you'd be correct to call it an exaggeration and somewhat toxic.

I mean, I find them underwhelming today. I just also haven't been given any convincing evidence that the technology is anywhere close to tapped out yet, and lots of evidence that it isn't.