As I understand it, "state" is actually just feeding in the history into the prompt as "context", so realistically this could help build history and context via lookups
eg.
Answer this as an assistant, blah blah blah.
You already said ${QUERY 1}, ${QUERY 2}. Refer to ${QUERY 3}, ${QUERY 4} for ideal formatting.
Prompt: "This is what the user asks chat gpt this round"
> It also would not create the non deterministic responses users have come to expect.
Probably slightly modifying text is less expensive than coming up with brand new text. So the text can be cached, and when retrieved, slightly altered before served. A "cached-but-not-really" cache
It also would not create the non deterministic responses users have come to expect.