I agree there is an issue here -- btw, you would notice when eventually the server key changes.

User friendly and secure-by-default clients will leverage the domain HTTPS CA to solve this (fetching the server key using https). The downside is that it will require d/l and install