I'm kind of wondering what the benefit is over having something like Yubikey at all instead of something that's just software when you just leave it in all the time.
Trust: a hardware token has a very limited interface where it can be attacked compared with a general purpose computer or phone.
Take the devices you use: how many exploits have there been in the last year where an attacker who could get you to click on a link, view an image, etc. could run code on the device? Maybe you have something like the iOS sandboxing between application which would stop a compromised browser from compromising your authenticator app but there are many cases where attackers have been able to bypass that.
Using a hardware token prevents almost all of those attacks and means that if you are compromised you'll have an easier time regaining control. They also have some nice benefits such as not running out of battery when you need them in an emergency.
I specifically use it to store my gpg/ssh keys on it. The keys are generated on the device and have never been in any computer's memory. The key answers challenges from an SSH server with the appropriate response. I do not want to insert something dangling off my key every time I do git push.
If your SSH private key is on the Yubikey then you will not lose your private keys. Even in the case of U2F, the attacker will not figure out your U2F private keys or even all the places you are registered.
His point, which is correct, is that you'll persistently lose access to your server anyways, because a backdoored SSH client is almost as bad as a compromised key. I use a Y4 for SSH, but it's good to be clear-eyed about the limitations.
In my opinion it's not worth the effort (and certainly not $50). It makes marginally +X harder to exploit yet marginally +X inconvenient to use = typical security through obscurity.
My bad, it's not what classic "through obscurity" means. Instead I meant something that makes exploitation more "obscure" (you need to be prepared to hijack a server vs simply leak the key).