This. Some standards bodies (arguably) made a big deal about client certificates some time ago to reliably pin client identities for client->server connections (whether it worked is a different story), and I certainly think having functionality for the reverse (pinning server identities) should exist too.

Doesn't have to be Gemini even, but I think getting buy in from browser vendors after the removal of HPKP is going to be a problem...