Which unfortunately tends to result in everything getting highlighted as if it was a string literal if you have any multi-line strings anywhere.

Yeah, I don't think there's a good way of highlighting part of a code file, what you're looking at depends on everything before and (to some extent) after it. You have to do the whole thing.

The 'extent' you talk about is exactly the lexer (parser) state, you just have to properly serialize it for the beginning of the buffer to get cheap redisplays. It's not rocket science but almost no editor got it right.

Yi is one editor that does incremental parsing correctly: https://github.com/yi-editor/yi