The domain that the verifier (the site trying to authenticate you) is at is part of the cryptographic process. If the domain doesn't match (ie you're at a phishing site) then the results of the cryptography won't be valid for the actual correct site, only the phishing site (which gets the phishing site nothing it can use).