Using the bandwidth gap between hidden states and tokens, this article offers one explanation for LLM engineering phenomena such as Chain of Thought, prompt length, few-shot learning, personality drift, and hallucinations.
A systematic tour of the entire LLM training pipeline—from data pipelines, scaling laws, system constraints, synthetic data, distillation, post-training, and evaluation systems to agent training—and how these mechanisms affect model selection, evaluation, and harness design for AI application engineers.
This article connects cross-entropy, perplexity, temperature, conditional entropy, hallucination detection, and prompt constraints into one line of reasoning, showing how entropy provides a unified language for understanding LLM training, inference, and product design. It also explains entropy’s practical value in RAG and engineering governance.
KV Cache is a key concept connecting Transformer theory with LLM engineering and deployment. Understanding it completes the path from how a model computes to how it runs.
Understand what LLM Chain-of-Thought (CoT) is and how prompt engineering can elicit Chain-of-Thought (CoT) from an LLM.
This article gave me—and may give you—a deeper understanding of the basic principles behind Transformers, replacing a black-box view of today’s mainstream LLMs. Cheers!