There’s no downside that I know of, but if you’re using keys there’s no upside, and if you’re not exclusively using keys there’s a huge downside to that.
The major downside of fail2ban is actually that it punishes you for using keys. If you have different keys for different machines and haven't configured your SSH client to pair them up you might attempt to login several times with the wrong key before getting in. You won't even notice this normally but fail2ban will trigger and ban you from the machine.
