An Overlooked Central Tension
If you have spent any time in AI engineering, you have probably collected a pile of scattered rules of thumb: Chain of Thought improves reasoning; a prompt should not be too long, or the model will overlook important information; few-shot examples need a consistent style; a model’s “personality” drifts during long conversations; and models sometimes invent facts with confidence. These phenomena appear independent, each with its own cause and remedy.
But that is not the case. These five phenomena—and many more variations encountered in engineering practice—are different expressions of the same mechanism. Once you understand that mechanism, the scattered rules suddenly connect into one clear picture. This article will give you that “master key” and use it to open five doors.
At the heart of the key is a phenomenon I call “the asymmetry between the model’s inner state and its mouth.” Let me establish it as briefly as possible.
The Shape of the Key: The Bandwidth Gap Between Hidden States and Tokens
When a model processes a sentence, all of its internal “thinking” occurs in a high-dimensional vector space. More specifically, at every position, each Transformer layer maintains a vector called a hidden state, typically with 4,096 or 8,192 dimensions. This vector is the true carrier of the model’s “inner state.” It contains its complete understanding of the current position, including semantics, context, possible options, stylistic tendencies, confidence, and all the other information intertwined within it.
For a rough estimate, a 4,096-dimensional hidden-state vector contains 32,768 bits of information capacity even if each dimension is represented with only eight bits. That is equivalent to all the pixel information in a small 64×64 grayscale thumbnail. This is the real bandwidth of the model’s “inner state.”
Now consider the model’s “mouth.” When the model outputs a token, it must select one item from its vocabulary. If the vocabulary contains 120,000 items, that choice conveys log₂(120000) ≈ 17 bits of information—roughly the amount needed to guess a five-digit password.
Put the numbers together and an astonishing fact appears: the model’s internal bandwidth is on the order of 32,768 bits, while its output bandwidth is on the order of 17 bits—a difference of almost 2,000 times. To generate every token, the model must compress its internal, high-fidelity “thumbnail” into a low-resolution “five-digit password” and send that outward.
This is the central dramatic conflict in LLM engineering: the model’s internal state is high-dimensional and rich, its mouth is low-dimensional and narrow, and the current architecture requires all thought to pass through the bottleneck of that mouth. Once you hold this key, many seemingly mysterious phenomena in LLM engineering reveal their true form. Let us begin opening doors.
Door 1: Why Chain of Thought Works
Chain of Thought, or CoT, is one of the most famous prompt engineering techniques. The method is simple: instead of asking the model for an answer directly, ask it to output its reasoning process before giving the answer. On many tasks, accuracy improves substantially—sometimes by as much as 20 percentage points.
But stop and consider how strange this is. The model’s own abilities have not changed at all. Its parameters, architecture, and training remain unchanged. Adding one sentence—“Let’s think step by step”—to the prompt causes accuracy to rise sharply. Where does the “additional ability” come from?
An intuitive answer might be, “Writing the reasoning process first lets the model think more clearly.” That is not wrong, but it treats CoT as magic instead of explaining the mechanism. Our key gives us a much deeper explanation.
When the model answers directly, it must complete all reasoning within one complete forward pass. In other words, within the fixed computational budget of, say, 80 Transformer layers, it has to understand the question, complete the arithmetic, and emit the answer. This is enough for a simple question, but not for one requiring multiple reasoning steps.
When the model outputs CoT first, something remarkable happens. Generating “Let us first list the known conditions” uses one complete forward pass; generating “From these conditions, we can infer” uses another; and so on. If the complete CoT contains 100 tokens, the model effectively undergoes 100 complete forward passes, yielding 100 times the total computation of a one-token direct answer.
What CoT really does, therefore, is not teach the model “how to think,” but provide a total computational budget far larger than one forward pass. It allows the model to divide a complex problem into multiple steps and process each with a complete forward pass. Every step still needs to squeeze through the narrow 17-bit channel of a token, but the accumulation of dozens or hundreds of steps carries enough total information to solve a complex problem.
This is why CoT does little for simple questions, where one forward pass is already sufficient, but helps significantly on multi-step reasoning, where one pass is not enough and a relay of steps is necessary. It also explains CoT’s hidden cost: it consumes many tokens, increasing both latency and expense. CoT fundamentally exchanges token count for computation, although the exchange is implicit and can look like a free lunch.
Once you understand this, the design of the thinking-model generation becomes clear. These models elevate CoT from an “optional prompt-engineering technique” into “built-in model behavior.” The model decides automatically whether to think, how much to think, and when to stop. The underlying mechanism is the same: accumulate multiple forward passes to exceed the computational limit of one step, at the cost of relaying every step through the narrow token channel.
Door 2: The Sweet Spot for Prompt Length
You have probably heard many recommendations about prompt writing: be detailed, be specific, provide examples, and define a role. These imply that a prompt should be long. But you also hear another set of recommendations: be concise, stay focused, and do not include too much information. These imply that a prompt should be short. Which is correct?
The answer is specific and can be explained clearly with our key.
When you give the model a prompt, it first converts the entire prompt through embeddings into a sequence of hidden states. The attention mechanism allows the hidden states at all positions to influence one another, eventually forming a hidden state at the final position that integrates the entire prompt. This final hidden state determines the model’s “frame of mind” when it generates the first output token.
When the prompt is short, this final hidden state can be “disorganized.” There is too little information for the model to know with confidence what it should do, which style it should use, or which direction it should take. Its internal state is distributed across several possibilities. Output generated from this state tends to be generic and insufficiently targeted.
As the prompt grows to an appropriate length, concrete instructions, examples, and constraints progressively “shape” the model’s internal state. The final hidden state becomes clear and focused. The model knows what you want, what you do not want, and which tone to use. Its output becomes specific and high-quality. This is the true mechanism of prompt engineering: using token input to sculpt the model’s internal hidden state, guiding that high-dimensional vector toward a position that produces high-quality output.
But if the prompt continues to grow, a problem emerges. Attention is a conserved resource: softmax makes all attention weights sum to one, so the longer the context becomes, the thinner each token’s share of attention becomes. Consequently, once a prompt is already sufficiently clear, adding more information no longer focuses the model’s internal state; it dilutes the strength with which each item reaches the final hidden state. The “voice” of essential instructions is drowned out by secondary information, and the model begins to ignore things that you believe you explained clearly.
This is why prompt length has a sweet spot. A prompt can be too short to shape the model, leaving its internal state too dispersed, or so long that it dilutes the key signals beneath secondary information. Between those extremes lies an optimal range. Its location varies by model and task, but finding it is one of a prompt engineer’s core responsibilities.
This understanding explains an apparent contradiction: sometimes deleting content from a prompt improves the result. This does not mean the model does not need that information; it means the information dilutes something more important. Excellent prompt engineering is not the endless addition of content but a trade-off between the value of every item and its dilution cost.
Door 3: What Few-Shot Examples Really Do
Continue using the same key. Few-shot prompting—giving the model several input-output examples in the prompt—is another widely used technique. The intuitive explanation is that examples “teach the model what to do.” As with CoT, this is not exactly wrong, but it does not reveal the mechanism.
Let us look again using our key. When you provide a few-shot example, its input becomes one sequence of hidden states and its output becomes another. The attention mechanism lets the model internally “sense” the correspondence between them. This correspondence is not explicitly extracted as “I learned a rule.” Instead, it directly biases how the model processes the hidden state of the next similar input.
More concretely, after several examples, the final hidden state for the actual input has already been pulled toward the style of the examples’ outputs. This is “anchoring” in vector space. The more numerous and consistent your examples are, the more strongly the model’s output hidden state is anchored in that direction, and the more its generated content resembles the examples.
This perspective explains several strange properties of few-shot prompting. First, diversity among examples weakens the anchor. If you provide three examples in very different styles, the model’s hidden state is pulled in three directions and lands somewhere near their average, weakening the anchor. Few-shot examples therefore normally need a consistent style. Second, the example’s format matters more than its content. Research has found that randomly scrambling the input-output labels in few-shot examples—for example, exchanging the labels for “positive sentiment” and “negative sentiment”—causes only a small decline in performance. The model primarily learns not “which input corresponds to which label,” but “what shape the input should have, what shape the output should have, and what kind of transformation should connect them.” These are structural properties at the hidden-state level and do not depend on the concrete semantic mapping. Third, the benefit of additional examples diminishes and may even reverse. This returns us to attention dilution: with too many examples, the anchoring strength contributed by each one declines.
Few-shot prompting is therefore not fundamentally “teaching” but “anchoring.” It draws a trajectory through hidden-state space with a token sequence and causes the model’s output to land on an extension of that trajectory. Once you understand this, designing a few-shot prompt changes from “What rule do I want to teach the model?” to “Where do I want to anchor the model’s output?” This is a completely different and more accurate operating framework.
Door 4: Why a Model’s “Personality” Is So Difficult to Keep Consistent
Moving further, we arrive at a deeper phenomenon: the consistency of a model’s “personality.”
If you converse with Claude for a long time, it seems to have a relatively stable personality. Its style of expression, values, and response rhythm have a certain coherence. But you may also notice that this coherence sometimes “breaks.” One answer suddenly becomes colder than usual, another becomes excessively enthusiastic, or on a particular topic Claude seems to become a different Claude. Many users have experienced this “personality drift.”
Why does it happen? Through our key, the answer becomes clear. Claude’s “personality” is not an entity explicitly stored somewhere; it is a stable attractor basin for hidden states in vector space. As a conversation proceeds, each turn’s hidden state is shaped jointly by all preceding tokens. If the conversation’s content, the user’s tone, and the topic remain consistent with “Claude’s typical personality region,” the hidden state remains stably within that region and Claude’s responses feel consistent.
But if one token pushes the hidden state in an unusual direction—for example, the user adopts an extreme tone, mentions a topic that activates unusual associations, or an early token in a long context repeatedly affects later states through attention—the hidden state departs from the stable attractor basin. Once it departs, the “tone” of Claude’s response changes with it, sometimes subtly and sometimes enough for a user to think immediately, “This does not sound like Claude.”
This is why the central mechanism of a jailbreak—causing a model to say something it ordinarily would not—is carefully constructing a token sequence that pushes the model’s hidden state into a region not covered by human feedback during training. In that region, the model still operates normally, but its output is no longer constrained by the “personality” shaped through human feedback. This is also why defending against jailbreaks is so difficult: it is impossible to enumerate every token sequence capable of pushing the hidden state into an anomalous region.
For an Agent engineer, this perspective has a highly practical implication. When you design an Agent system, your prompts—SOUL.md, AGENTS.md, and similar files—do one thing: anchor the Agent’s hidden state in a stable personality and capability region. The better the prompt, the stronger the anchor and the lower the probability that the Agent will “drift” during a long conversation. But this is never 100% reliable, because subsequent tokens can always push the state away. A robust Agent design should therefore assume that drift will occur and create monitoring and regression mechanisms—for example, periodically injecting a “review instruction” to recalibrate the hidden state, or reloading the complete system prompt at critical decision points.
Door 5: Why Models Hallucinate
The final door leads to the phenomenon that frustrates LLM engineers most: hallucination.
A hallucination occurs when a model confidently states something factually incorrect, such as inventing a nonexistent book, fabricating a person’s biography, or supplying an incorrect historical date. The intuitive explanation is usually that “the model does not know what it does not know, so it makes something up.” That explanation is half right but misses the most important part of the mechanism.
Consider it with our key. When you ask, “Who wrote the book XXX?”, the model performs a forward pass, produces a hidden state in its final layer, and decodes output tokens from that state. The central issue is that the structure of hidden-state space determines which outputs “look plausible.” In this high-dimensional space, the hidden states corresponding to “The author is A” and “The author is B” are normally very close. Both satisfy constraints such as “the answer is a person’s name,” “the name appeared in a context related to this book,” and “the name lies within the semantic cluster of literary authors.”
If the model encountered the book’s real author during training, the corresponding hidden state for that author is reinforced and the model generates the correct answer. If the model never saw the book, or saw it too rarely to establish a strong signal, the output “The author is X” is decoded from a temporarily synthesized hidden state whose form is plausible but whose content is empty. The model is not “inventing at random”; it generates a plausible-looking output that satisfies the constraints of the space, but that output lacks a factual foundation.
At a deeper level, while producing the output, the model has no mechanism that tells it whether the name it just generated comes from a real memory or from synthesis. Hidden-state space has no independent “factual/synthetic” dimension. This is a fundamental blind spot in the current Transformer architecture. Its internal state does not distinguish factual memory from plausible guessing; the two look the same in hidden-state space. The model’s “confidence” in an output comes from the clarity of its hidden state, not from whether a fact corresponding to the output actually appeared in its training data.
Once you understand this, it becomes clear why the simple instruction “say you do not know when you are uncertain” is insufficient. At the hidden-state level, the model does not necessarily have that “I am uncertain” signal. Techniques such as RAG, tool use, and grounding fundamentally avoid the impossible task of asking the model to determine truth by itself and instead supply factual constraints externally. This is the most practical current defense against hallucinations, and it works precisely because it diagnoses the real mechanism correctly.
Connecting the Five Doors
Pause and look back at the five doors we have opened: why CoT works, the sweet spot for prompt length, what few-shot examples truly do, the stability of a model’s personality, and the origin of hallucinations. On the surface, these are five separate topics in LLM engineering. But you should now sense that they are different expressions of one underlying mechanism.
That mechanism is the central tension established at the beginning: the model’s inner state is rich, high-dimensional cognition in hidden-state space; its mouth is the narrow, low-dimensional channel of a 17-bit token; and every LLM engineering practice fundamentally manages the bottleneck in transforming one into the other.
CoT uses a multi-step relay of tokens to accumulate total computation and work around the limit of one forward pass. The sweet spot for prompt length balances “enough information to shape the hidden state” against “not so much that key signals are diluted.” Few-shot examples draw a trajectory through hidden-state space and create an anchor. A model’s personality is a stable attractor basin in hidden-state space. Hallucination is a byproduct of the structure of that space because the space does not distinguish “true” from “plausible.”
That is why understanding the central tension gives you a master key. Whenever you encounter an LLM engineering problem in the future, your first response should be to ask: What is happening at the hidden-state level? What is happening at the token level? Where did the transformation between them fail? This framework lets you look one layer deeper than most prompt engineers. Many stop at trial and error over tokens; you can reason at the hidden-state level.
An Exercise to Internalize the Framework
Finally, here is a thought exercise to turn this framework into engineering intuition.
Recall a recent piece of LLM-related work: perhaps designing an Agent Skill, debugging a prompt, or modifying a system prompt. Choose one concrete design decision or problem and analyze it again with the “hidden state versus token” framework developed here. What did that decision actually affect at the hidden-state level? How did it relate to behavior at the token level? If you redesigned it, could a hidden-state perspective reveal a better approach?
The purpose is not to reach a new conclusion immediately, but to build the habit of thinking with this framework. After several such exercises, you may find that your perspective on LLM engineering problems has changed permanently. This shift may be one of the most valuable ways for an AI engineer to differentiate themselves today. Most practitioners, even those with solid technical skills, rarely explain LLM behavior at the mechanism level. If you can, your technical depth will stand out clearly among peers with comparable experience.
This key will not become obsolete. Even if the Transformer architecture is replaced by something more advanced, as long as an AI system distinguishes an “internal representation” from an “external interface,” there will be an asymmetry between its inner state and its mouth, and some variation of this central tension will remain. Mastering this framework gives you not only a tool for explaining today’s LLMs, but a general mental model for every system in which internal state and external interface diverge.