The rigorous form of the non-rigorous version is non-standard analysis: There really are tiny little numbers we can manipulate algebraically and we don't need the epsilon-delta machinery to do "real math". It's so commonsensical that both Newton and Leibniz invented it in that form before rigor became the fashion, and the textbook "Calculus Made Easy" was doing it that way in 1910, a half-century before Robinson came along and showed us it was rigorous all along.
> The rigorous form of the non-rigorous version is non-standard analysis
This is quite overstated. There are other approaches to infinitesimals such as synthetic differential geometry (SDG aka. smooth infinitesimal analysis) that are probably more intuitive in some ways and less so in others. SDG infinitesimals lose the ordering of hyperreals in non-standard analysis and force you to use some non-classical logic (intuitively, smooth infinitesimals are "neither equal nor non-equal to 0", wherein classical reasoning would conflate every infinitesimal with 0), but in return you gain nilpotency (d^n = 0 for any infinitesimal d) which is often regarded as a desirable feature in informal reasoning.
https://calculusmadeeasy.org/
https://en.wikipedia.org/wiki/Calculus_Made_Easy