Huh. "Nucleus" is a form that looks like an adjective to me; I'd expect "nucula" for the diminutive of "nux".

And indeed I see this on etymonline:

> from Latin nucleus "kernel," from nucula "little nut," diminutive of nux (genitive nucis) "nut,"

So for whatever reason a new word was called for, and our nucleus comes from that, not from the diminutive form of "nut".