Not listening on port 80, such that the user gets a connection refused, would result in the client not sending the api key over the wire at all.

I personally think listening, accepting that user mistakes can expose API keys to MITMs, and returning the user-facing error is better than a "connection refused" error, but it is a tradeoff.