https://sinimite.work/en/tags/informationtheory/Recent content in InformationTheory on SiBloghttps://sinimite.work/images/og-default.svg?v=20260525-210321https://sinimite.work/images/og-default.svg?v=20260525-210321Hugo -- 0.156.0en-USFri, 27 Mar 2026 00:00:00 +0000https://sinimite.work/en/posts/entropy-in-llm/Fri, 27 Mar 2026 00:00:00 +0000https://sinimite.work/en/posts/entropy-in-llm/Entropy runs through the entire LLM lifecycle: training uses cross-entropy as the loss, inference uses temperature to control output entropy, evaluation uses perplexity, and products can use entropy signals to detect hallucinations. This article maps the complete picture from an AI engineer's perspective.