You shouldn't be logging in to a server via SSH using a user+password combo, instead use a public/private key combo which is considerably more complex and can't effectively be bruteforced like a user+password.
Most web servers don't really come with any built in defense against brute force attempts vs Basic Auth gates, so unless you've set something up to protect it, someone with enough time will eventually get in.
Genuine question that I haven't found a good solution to yet, if I want to just go to any old computer and ssh into my server, do I have to carry around a USB stick with the ssh key on or something? because I sure as hell wont be able to just remember it
In that case I'd normally recommend a bastion host with SSH MFA and fail2ban. It'd be publicly available and have SSH keys for other machines. Or you could look at setting up a VPN solution with MFA, but never have a password only admin login exposed to the public Internet.
Most web servers don't really come with any built in defense against brute force attempts vs Basic Auth gates, so unless you've set something up to protect it, someone with enough time will eventually get in.