Concepts vs. Associations: Why LLMs Don't Rank Your Goal Above Other Text
Updated
Knowledge on this page was mainly distilled from Do LLMs Understand Context? Context Rot Doesn't Explain Why Your Goals Get Ignored.
People and language models hold a project's goal in different forms. A person holds a project as a small hierarchy of ranked concepts, and that hierarchy frames every problem before it is even considered. A model holds the same goal as text, weighed against everything else in its context window by how well it matches the current request. Nothing in the model ranks the goal first for the task at hand, so the goal can be fully present and still lose.
How people hold a project
I describe thinking about a project not as holding the documents that describe it, but as holding a few concepts, each usually labeled with one word. The reasoning that produced each concept did its job once and does not need rereading.
Psychology shows why this works. In 1973, William Chase and Herbert Simon showed chess positions to players for five seconds and asked them to rebuild the board from memory. A master rebuilt positions from real games far better than weaker players, but when the same pieces were placed at random, the advantage disappeared. The master stored meaningful groups of pieces as single units, and a random board has no such groups to find.
Compression is necessary because the number of units we can hold at once is small. Nelson Cowan's review puts it at about four; I think it is closer to 5-7. Either way, nobody holds a whole product in mind as sentences.
Those concepts also form a hierarchy. In my app, what the app is for sits at the top, "overwhelm" sits below it, and every screen and feature that might be changed to reduce it sits below that. For a given task, he also knows which concept matters most. So a concept does not sit beside a problem waiting to be consulted. It reshapes the problem: "reduce overwhelm" is read as "reduce overwhelm for people who came here to do this particular thing."
How a model holds the same goal
A model has associations learned in training: which words, ideas and patterns tend to go together. These have some structure. Probes detect simple distinctions in early layers and complex ones only in deeper layers, and representations follow category trees such as animal, mammal, dog. But that structure comes from training, and none of it ranks which idea matters most for the task in front of the model.
Inside the model, weight follows match. For each word it writes, the model compares its current position, shaped by the request, with the stored record of every earlier token, and stronger matches count more. My intent file described what his app is for, while his request about overwhelm was about screens, steps and clutter. A request about clutter matches everything else about clutter and barely matches a statement of purpose. The model's suggestions were sensible, and some cut the very things the app exists to do. No association told it the purpose should win anyway.
Having context versus having your text
This is where the word context misleads. When a colleague has context on your app, they know what it is for and what that rules out before you ask anything. When a model has context, it has your text: the intent file, the request and everything else, side by side. In my reading, the model never forgot the goal. It never held the concepts in the intent file as concepts, and it never knew which of them comes first. Whether the goal shapes an answer is decided again every time the model produces words.
Q&A
Can LLMs understand context at all?
They can seem to use it, often very well. What they do not do, in my view, is turn a project's goal into a concept that frames every answer. The goal stays text, and each answer decides again how much that text matters relative to everything else in the window.
Why did the chess masters lose their advantage on random boards?
Their advantage came from what they stored, not from raw memory capacity. Masters recognized meaningful groups of pieces from real games and held each group as a single unit. Randomly placed pieces form no such groups, so masters had nothing to compress and performed like weaker players.
Does the exact working memory limit matter for this argument?
Not much. Cowan's review puts the limit at about four units, and I suspect it is closer to 5-7. Either number is far too small to hold a product as sentences, which is why people compress experience into a handful of concepts, each standing for more than they could recite.
Can context engineering make a model treat the goal as a concept?
I argue it cannot. You can insist through context engineering that the model treat your goal as a framing concept, but the goal remains text weighed by associative match. Any instruction you add is also text and is subject to the same matching against the request.