It's complicated. There's a fixed amount of design overhead in supporting all the various x86 modes but compared to designing a top-end processor those aren't too big. And being able to decode 4 x86 instruction at once is hard compared to doing that with ARM but compared to the size of a 192 entry reorder buffer for deep out of order execution the cost isn't huge. Both x86 and ARM want to convert their ISA operations to a different format for use internally. ARM's is closer to their ISA instructions but it's not clear that that makes a big difference. The biggest issue in practice at the high end might memory ordering constraints with x86 being very tight and ARM being very loose. But I can't actually say for sure.

But this is probably why Intel's efforts to extend x86 down to lower power processors haven't been huge successes.