If you're going to create a wildly incompatible "better x86", just start with a RISC design and be done with it. Compatibility with modern x86 OS's is the reason for x86's existence (note that v8086 mode is not required for this). Drop that and you might as well just fix all the other problems with the ISA while you're at it.
I'd be interested to know how much ARM have an advantage here in terms of their ISA? Presumably if more resources were dedicated to it, ARM cpus could be quite a lot faster than x86?
I doubt it. For one, 32-bit ARM is pretty complex too (though 64-bit ARM is a very nice simplification). More importantly, though, you can overcome the x86 complexity by simply throwing enormous manpower at the problem, and that is what Intel has done over the years.
(That doesn't mean the complexity is justified, just that it can be overcome. I think that people often assume that x86's dominance means that there's something inherently amazing about x86 from a microarchitectural point of view. That isn't true; rather it's simply that x86/Windows has historically earned Intel so much revenue that they've been able to fund the manpower needed to keep it alive and on top.)
It's complicated. There's a fixed amount of design overhead in supporting all the various x86 modes but compared to designing a top-end processor those aren't too big. And being able to decode 4 x86 instruction at once is hard compared to doing that with ARM but compared to the size of a 192 entry reorder buffer for deep out of order execution the cost isn't huge. Both x86 and ARM want to convert their ISA operations to a different format for use internally. ARM's is closer to their ISA instructions but it's not clear that that makes a big difference. The biggest issue in practice at the high end might memory ordering constraints with x86 being very tight and ARM being very loose. But I can't actually say for sure.
But this is probably why Intel's efforts to extend x86 down to lower power processors haven't been huge successes.
