Fixing WordPress's code problems don't really need to break plugin compatibility. They can rewrite the core, have new features requiring doing things the new way, and have small adapters to maintain the backward compatibility.
But yes, this refactoring is going to cost and there is unfortunately not many people for whom it is a worthwhile expense. If Automattic is able to think about 10 years in the future, it might make sense for them to start investing in such a clean up.
It's a very tough act to balance, because too many changes will prompt criticism that all kinds of things are breaking on a new update. You can't please both ways, so things move pretty gradually.
Having said that each new version has improvements. The direction I think is more dictated towards where the web is going. Each new release increases the amount of javascript, even backbone has made its way into core. There will also soon be better support for other DBs.
But yes, this refactoring is going to cost and there is unfortunately not many people for whom it is a worthwhile expense. If Automattic is able to think about 10 years in the future, it might make sense for them to start investing in such a clean up.