You could enter a password using the Yubikey instead, provided there's a way to input a password. That might make brute-forcing a little challenging especially if you added tamper evidence and heat-sensitive fuses that blow if someone tries to open the case and modify the device. At that point you might be stuck brute forcing with a pen plotter.