“I'm not sure how nuclear armament is going to end, but one thing I do know is that I don't want a small number of giant countries to hold the reins.”

Perhaps you think this analogy is a stretch, but why are you sure you don't want power concentrated if you aren't sure about the nature of the power? Or do you in fact think that we would be safer if more countries had weapons of mass destruction?

I would feel very uncomfortable if the companies currently dealing in AI were the only ones to hold nukes.

Not sure if this answers your question.

information != nukes

One directly blows people up, the other gives humans super powers.

Giving individual people more information and power for creativity is a good thing. Of course there are downsides for any technological advancement, but the upsides for everyone vastly outweigh them in a way that is fundamentally different than nuclear weapons.

Empirically, countries with nuclear weapons don't get invaded, so in that sense we'd expect to have seen fewer wars over the past few decades if more countries had nukes. Russia would probably never have invaded Ukraine if Ukraine had nukes.

The analogy would be corporations controlling the weapons of mass destruction.

Sure. I would feel much safer if only FAANG had nukes than if the car wash down the street also had one.

I want my government to have them (or better, nobody), not FAANG or car washes.

With open-source models, this is just a dream. With closed-source models, that could eventually become the de facto state of things, due to regulation.

Comparing this to nuclear weapons is laughable.

Is it still a "laughable" comparison if AI systems eventually become smart enough to design better nukes?

Yes it is. You can build a bomb many times more powerful than the bombs dropped on Hiroshima and Nagasaki with publicly available information. If the current spat of ai bullshit knows how to build a bomb, they know that because it was on the public internet. They can never know more.

The hard part of building nuclear bombs is how controlled fissile material is. Iran and North Korea for example know how to build bombs, that was never a question.