The file format doesn’t matter one bit when the reading and authoring tools are shit and the editors can’t/don’t fix anything. And papers will generally have a lot fewer resources to deal with this than major book publishers, who have been epub-focused for over a decade now and actually make money from it.

Any decent publishing or html editing tools fully support utf-8 by now. It's not the tools.

Publisher and editor laziness may be a reason to be cautious about epubs currently for niche or esoteric works, but that's not the same thing.

> I bought a book set in the late Middle Ages which managed to transcribe all “þ” as “p”. Until publishers care...

The book market these days makes it challenging to do high-quality editing up front for republishing niche books in a new format. Publishers try to cut corners, outsourcing epub conversions to people who don't care and don't know what they're doing, or they OCR it, have an in-house editor (who also doesn't have a personal affinity to the subject) give it a once-over (maybe), and release it.

As an aside : Unicode support was still an issue in TeX last I checked, because most of the LaTeX tools don't support it (well, having been made before it was expected).

Now, there are some attempts to fix this situation by Xe(La)TeX and Lua(La)TeX, but since TeX seems to be so much tied to PDF these days, it should probably just be abandoned by most scientific publishing in favor of the likes of GNU TeXmacs (note : it's NOT TeX in GNU Emacs) and HTML with MathML.