So build one. Your preference actually has very little to do with the X509 CA architecture. Today we have a complex little forest of fiat CAs. No part of the TLS architecture prevents you from replacing that with an even more complex and more full-featured web of smaller CAs.

I'm not being sarcastic. Your decentralized reputation scheme could very well be better than our severely compromised system of central CAs.