This seems like a good flow! I end up adding a "spec" and "todo" file for each feature[1]. This allows me to flesh out some of the architectural/technical decisions in advance and keep the LLM on the rails when the context gets very long.
Yeah, I limit context by regularly trimming the TODOs. I like having 5-6 in one file because it sometimes informs the LLM as to how to complete the first in a way that makes sense for the follow-ups.
READMEs per module also help, but it really depends a lot on the model. Gemini will happily traipse all over your codebase at random, gpt-4.1 will do inline imports inside functions because it seems to lack any sort of situational awareness, Claude so far gets things mostly right.
[1] https://notes.jessmart.in/My+Writings/Pair+Programming+with+...