The question I would like to know is whether that just leads you back to hallucinations. ie: is the avoidance of hallucinations intrinsically due to forcing the LLM to consider limited context, rather than directing it to specific / on topic context. Not sure how well this has been established for large context windows?
Having details in context seems to reduce hallucinations, which makes sense if we'd switch to using the more accurate term of confabulations.
LLM confabulations generally occur when they don't have the information to answer, so they make it up, similar to it you've seen split brain studies where one hemisphere is shown something that gets a reaction and the other hemisphere is explaining it with BS.
So yes, RAG is always going to potentially have confabulations if it cuts off the relevant data. But large contexts themselves shouldn't cause it.
