What if the server doesn't support version X? Then the client will try again with version X-1. This is a negotiation, it is just an inefficient one (client might have to try X-1, X-2, X-3 in turn if more versions are still co-existing; and contrarily, if client doesn't support any version the server does, you will not get a detailed error about the version mismatch, because they are entirely different protocols).
Who says that? Either it is specified in the protocol, and then you're right that this is still a negotiation, or it is not specified in the protocol, so if the client does that is implementing some out of spec behavior. Which is true that already happened on the Internet for compatibility's sake, but keep in mind that there is still an advantage, as an adversary can't easily mess in the connection setup phase in a way that protocol version X looks like protocol version Y.
If you don't have this downgrade, then clients can't talk with servers that haven't been upgraded yet! This means either that:
* you expect all servers to upgrade immediately, or at least faster than clients, or
* you delay upgrading clients you know to be insecure until servers had time to upgrade, or
* you are ok with breaking a significant portion of the network for every protocol update.
Either way, your proposal makes no sense for the internet.
Additionally, this is already possible with the current negotiation scheme. You can have clients refuse old algorithms (and they do). Your proposal doesn't improve anything there or anywhere else.
Negotiation makes the protocol more complicated, so more room for bugs. Additionally, sometimes (or maybe all times?) you need to explicitly configure a client or server to not accept broken algorithms during the negotiation, so you would need to keep track for yourself for which algorithms are safe to use and which are not. Anyway it would be interesting to look at how the Wireguard folks would handle this, as Wireguard is a protocol with opinionated crypto.
