Would banning any IP that tries connecting on 22 increase security?

You could do something like that, but you'd have to write it yourself. If you want to ban IP addresses, fail2ban is probably an easier starting point. Plus users who authenticate with passwords might occasionally make typos and get banned, incentivizing better logon techniques.

For my random high numbered IP ssh port, I see 5-15 attempts per day. In the last 3 months, the most failed attempts from a single IP is 3. Most attackers (at least amongst those that try higher ports) are distributed these days.

how do you connect rightfully then? at that point is easier to have and access rule on the firewall and only accept connections from the allowed ips.