An attacker would not bother to brute-force non-22 SSH ports.

The mere fact that you bothered to chang the port is indication that you likely are not using common passwords either.