Look, 2048 is fine. Assuming no algorithmic speeduos you get 112 bits of security and that’s plenty. But SSH keys are only used to sign: they don’t affect bulk encryption. 4096’s performance is not what’s holding you back.
"Slows down" can also mean connection times. On a computationally weak device doing 4096-bit RSA is far from instant. This is, after all, one of the reasons people are enthusiastic about Elliptic Curve options in this space.
Some people need SSH to move a lot of data, e.g. for SFTP but some people just want their connection to a nearby machine to feel "snappy" and not take a beat to do the key exchange and authentication steps.
I'm not thinking about myself, the desktop I do Social Media from can do almost three hundred, and indeed RSA 4096 seems fine on that PC, but lots of people have crappy under-powered devices like Raspberry Pis. How many can those do? Four? Ten?
We're in the weeds here, we're agreed that if your weakest point is a 2048-bit RSA key you're in unexpectedly good shape, definitely anyone who feels 4096 even "might be" too slow should just use RSA 2048 (or get an elliptic curve algorithm that's nice and fast on their CPU). I was just pointing out that "too slow" doesn't necessarily mean "Not as much peak throughput as I would like". Station wagons full of tapes remain sub-optimal for video conferencing :D
I do recollect that using RSA keys of 4096 bits slows down SSH more than the gain in security might be worth.