That would be a disaster of epic proportions. Every thread, no matter how innocuous the topic, would devolve into perpetual feuds about which kinds of people were upvoting and downvoting which other kinds. These sorts of shitstorms were pretty common on HN, in the long-long-ago, and even then all we were revealing were comment scores, not voting attribution.
Then we remove upvotes and downvote entirely. Web2.0 online discourse (visibility is controlled by recommendation algos keyed on upvotes/downvotes) is fundamentally broken.
That would create a different forum. HN is an experiment in how long this kind of forum can ward off gravitational collapse. There are things fundamentally wrong with these kinds of forums, but that's true of every kind of forum.
Not to pile on what tptacek has already nicely said, but I agree and still think there is plenty of room for new forums. The vast majority of possible permutations have not been tried yet, and I wish more people would.