What Context Actually Means to an AI Model

"Context" gets used for a prompt, a chat history, a retrieved document set, and a vaguer idea of "what the model knows about the situation." Mechanically there is only one thing: the sequence of tokens present in the model's input at the moment it generates a response. Nothing outside that sequence has any effect on the output — not a fact that's true, not a document that exists somewhere, not a decision the company made last week. If it isn't in the tokens, it isn't context, regardless of what a dashboard elsewhere in the stack believes.

It reframes every claim about an AI system "knowing" something as a claim about a pipeline: something upstream of inference decided which tokens to include, in which order, truncated to which length. "The assistant knows our refund policy" really means "a retrieval step, a system prompt, or a fine-tune put refund-policy tokens in front of the model recently enough to still be there." Change any part of that pipeline and the model's apparent knowledge changes with it, instantly and completely, because nothing was ever stored inside the model between calls.

Once context is understood as assembled input rather than accumulated knowledge, the real problem becomes selection and freshness: out of everything a company could supply, which tokens are relevant to this specific request, and are they still accurate. Get selection wrong and the model either lacks what it needs or drowns in irrelevant text competing for the same limited window. Get freshness wrong and the model states a superseded fact with the same confidence as a current one, because token sequences carry no timestamp the model can reason about on its own.

This is the problem a memory layer is actually solving — not giving a model a bigger window, but deciding, correctly and current as of right now, which tokens belong in whatever window it has.