First of all the web isn't only HTML. I can put a plain text or a PDF document or a big gif and it will serve up and display just fine. The reason why people don't just do that is that said document formats aren't designed to solve the problems that HTML already does - problems such as how to display reasonably on a wide variety of devices.

But suppose we wanted to create a new format that does what HTML does, only better in some way.

There is a hard to solve chicken and egg problem here.

Our format is no good until we have created and deployed browsers how to read it. There is no motivation to do that when nobody has such documents. (Note that the other formats that I just named predated the web so there were a lot of documents available from the beginning.) There is no motivation for a website to put such a document up when nobody has browsers that can read it.

And if we solve that problem, that's when we begin to solve the problem of everyone learning how the format works, and how to make it do what we want to do in a wide variety of circumstances. There is a tremendous amount of industry knowledge about HTML, and it takes time to get people to learn the equivalent about the new format. And until they do, what is the motivation for businesses to jump into the new thing that doesn't really work yet?

The web isn't HTML-based only: it is increasingly Javascript based. In fact, there's a surprising number of sites posted to HN whose entire HTML is a reference to a script that requests some Markdown to render.