Emacs in a way does something like this: just about any emacs app (including mail clients) can run in either terminal emacs or GUI emacs, and the GUI version supports proportional fonts, images, etc. Of course building your TUI app on top of emacs instead of ncurses isn't for everyone.
i had similar thoughts. what would a modern terminal look like? one where positions and cursor movements are not defined by obscure escape codes but by a proper API.
Immediate advantages, at least for me:
- a possibility to use proportional fonts, it's more pleasant for me and increases content density
- borders and other decorations could be smaller, increasing density
- smooth scrolling could be implemented, which is nice
- it could better integrate with desktop environment
Possible and very much desired extensions:
- basic image and video support
- possibility to use different font sizes
It would still be limited to a coarse grid, but that's what makes development easy.
I never looked beyond superficial it's possible or easy enough to do. I also don't have much experience, beyond basics with ncurses.