How to Design an Excellent AI Agent: From Architectural Principles to Practical Patterns

A complete guide spanning Software Specs, a three-layer architecture, the Latent-versus-Deterministic boundary, and self-improving systems, based on an in-depth reading of leading AI coding agent practices from 2025–2026.

Introduction: In the AI Agent Era, the Engineer’s Role Has Changed

In 2025–2026, AI coding agents underwent a qualitative change. Claude Code, Codex CLI, Cursor, and Kiro are no longer autocomplete tools that merely complete the next line. They have evolved into autonomous agents that understand an entire repository, modify multiple files, run tests, and iterate independently. Google’s Addy Osmani calls this new mode “AI-augmented software engineering”—software engineering enhanced by AI rather than automated by it.

The distinction is critical. The developer’s role is shifting from “the person who writes code” to “the person who directs AI to write code.” In a widely circulated slide, Andrew Ng highlighted a deeper consequence: after AI coding dramatically accelerates the “building” stage, product management becomes the new bottleneck. Engineers can produce three prototypes in a day, but product managers cannot provide feedback quickly enough. His conclusion is that the Engineer-to-PM ratio is moving toward 1:1, and engineers who can shape product direction themselves will move extremely quickly because they do not have to wait for anyone; they can complete the full build → feedback → iterate loop independently.

Thus, the most valuable engineers in the AI era are not those who type code fastest, but those who can act as both an engineer and half a product manager. The technical vehicle for that ability is the central subject of this article: how to design an excellent AI agent system.

1. Begin with the Spec: The “Contract” Between You and AI

Before architecture, there is a more basic question: how do you tell AI what you want?

1.1 A Software Spec Is Not a Prompt

In the context of AI coding agents, a Software Spec is essentially a requirements contract written for AI. It precisely describes what you want, what you do not want, and where the boundaries lie, enabling the AI to produce code close to your expectations with minimal back-and-forth.

But a Spec is not a prompt. A prompt is one message that disappears into chat history. A Spec is a persistent, version-controlled document anchored in the repository, surviving across sessions and evolving with the project. In September 2025, GitHub open-sourced Spec Kit and formally introduced Spec-Driven Development, or SDD. A clear consensus is emerging: in the AI agent era, the specification is replacing code as the source of truth.

1.2 Why Specs Have Become More Important Than Ever

An AI coding agent is your “super junior engineer”: extraordinarily fast and capable of execution, but unwilling or unable to infer your intent reliably. When a Spec has gaps, the AI may not stop to ask; it fills them itself, often in ways you did not want. Ask it to “write a user authentication module,” and it may build JWT, OAuth2, and social login when you only needed API-key validation. Overengineering is AI’s default mode. The better the Spec, the closer the output is to the picture in your mind.

More importantly, research shows that placing too many instructions in a prompt significantly reduces the model’s compliance with each one. This is called the Curse of Instructions. You cannot pile every requirement into one message; you need a structured, layered Spec to manage the information.

1.3 A Good Spec Covers Six Areas

After analyzing more than 2,500 agents.md files, GitHub’s AI team found that the most effective specs cover six areas: Commands—complete executable commands with flags and arguments, placed early in the Spec; Testing—how to run tests, the framework, and coverage requirements; Project Structure—explicit purposes for every directory; Code Style—one real code example is more effective than three paragraphs; Git Workflow—branch naming and commit-message format; and Boundaries—things the agent must never touch.

Boundaries are the easiest control to overlook and one of the most effective. Reworking something the AI added but you did not want is much more costly than asking it to add something it omitted. A rule such as “do not introduce dependencies absent from the current requirements.txt” can save two hours of investigation.

1.4 The Spec Workflow: A Four-Stage Gate

GitHub Spec Kit proposes a field-tested four-stage workflow. The core idea is to place a human checkpoint at every stage and proceed only after the current stage is verified:

Specify → Plan → Tasks → Implement

This gated workflow solves the problem of house-of-cards code: fragile AI output that appears to run but collapses under pressure. Instead of reviewing a thousand-line diff, you inspect small changes that solve specific problems. The Agent knows what to build from the Spec, how to build it from the Plan, and what it is currently doing from the Task.

2. The Three-Layer Architecture: Fat Skills, Thin Harness, Deterministic Tooling

After the Spec, the next question is how to design the AI agent system itself.

One strong architectural model divides the system into three layers and states a concise principle: push intelligence upward into the skill layer, push execution downward into deterministic tooling, and keep the harness in the middle thin.

2.1 Bottom Layer: Deterministic Tooling

The bottom layer contains deterministic capabilities exposed by your application: QueryDB, ReadDoc, Search, Timeline—SQL queries, document reads, search execution, and timeline retrieval. Their defining characteristic is that the same input always produces the same output. They need no “intelligence”; they only need exact execution.

This layer is the familiar territory of traditional software engineering. If you have Java backend experience, it resembles the DAO layer and infrastructure code you used to write: API calls, database queries, and file operations. Its value lies in reliability and predictability.

2.2 Middle Layer: Thin Harness

The middle layer is the outer program that runs the LLM—the shell behind tools such as Claude Code and Codex CLI. It does only four things: run the model in an agentic loop, read and write files, manage the context window, and enforce safety policy. Nothing more.

The core logic of a minimal agentic loop looks roughly like this:

loop {
    collect context (user input + system instructions + tool results)
    call the model
    if the model requests a tool → execute the tool and continue
    if the model returns a final answer → output and stop
}

Together with context management, safety policy, and error handling, this core loop can be implemented in roughly 200 lines. The crucial point is that the harness contains no domain knowledge, which belongs in skills; no concrete tool implementations, which belong in the application below; and no business judgment, which belongs in the LLM’s latent space.

In Java backend terms, the harness resembles the Spring Boot framework itself. You would not mix DispatcherServlet routing logic with business code. The framework schedules work, the business layer implements logic, and the separation is clear.

The anti-pattern is a fat harness with thin skills. You may have seen 40 or more tool definitions consume half the context window; every MCP tool call adds a 2–5 second network round trip; every REST API endpoint is wrapped as an independent tool. The result is three times the token use, three times the latency, and three times the failure rate. The correct approach is the reverse: fat skills, a thin harness, and tools that are specialized and fast.

2.3 Top Layer: Fat Skills

The top layer consists of workflows written in Markdown that encode judgment, process, and domain knowledge. The author argues that 90% of the value lies here.

“Fat” does not mean a long skill file. It means a skill carries high knowledge density. A good skill file precisely encodes which decision to make under which conditions, what process to follow, and which domain knowledge to apply. For example, a last30days skill’s SKILL.md can describe a complete multiplatform data-collection process: which API to use for each platform, how to expand queries, how to score and rank results, and which output format to produce. These are natural-language instructions rather than code, yet they encode the real expertise.

2.4 The Architecture’s Most Elegant Property

“When you do this, every model improvement automatically improves every skill, while the deterministic layer remains perfectly reliable.”

Because a skill expresses judgment in natural language, upgrading the underlying model from Sonnet to Opus or from one generation to the next automatically improves execution of the judgment encoded in the skill. A stronger model understands and follows the same instructions better, without a code change. Meanwhile, deterministic tools remain unaffected by the model replacement. A SQL query remains a SQL query regardless of which model runs above it.

The system’s two halves evolve differently: the upper layer improves automatically with the model; the lower layer remains perfectly stable. Their combination compounds.

A precise traditional-backend analogy is: the Controller layer, or harness, stays thin and handles scheduling; the Service layer, or skills, is fat and contains business logic and judgment; and the DAO layer, or deterministic tooling, performs deterministic data operations. The structures are isomorphic, except “business logic” becomes “judgment” and “code” becomes “natural-language instructions.”

3. Resolver: The Routing Table for Context

As your skills and domain knowledge grow, a new problem appears: how does the model load the right knowledge at the right time? This is the problem a Resolver solves.

3.1 Core Concept

A Resolver is a routing table for context. For a task of type X, load document Y first. A Skill tells the model how to work; a Resolver tells it when to load what. They operate at different levels.

For example, a developer changes a prompt. Without a resolver, the change goes directly to production. With a resolver, the model first reads docs/EVALS.md, which says to run the evaluation suite after a prompt change, compare scores, and roll back and investigate if accuracy drops by more than 2%. The developer may not even know the suite exists. The resolver loads the right context at the right time.

3.2 The Fundamental Tension Resolvers Solve

AI agent architecture has a fundamental tension: the more the model knows, the better; the fuller the context window, the worse it performs.

A real example illustrates this perfectly. One developer’s CLAUDE.md grew to 20,000 lines because every quirk, pattern, and lesson learned was added to it. The model’s attention degraded, and Claude Code itself suggested reducing the file. The developer eventually cut CLAUDE.md to roughly 200 lines containing only pointers—an index—to other documents, letting the resolver load them on demand.

The 20,000 lines of knowledge still exist; they no longer pollute the context window. This is lazy loading. You do not load an entire database into memory at application startup; you query a row only when needed. A Resolver is lazy loading for the context window.

3.3 Claude Code’s Built-In Resolver

Claude Code has a built-in resolver mechanism: each skill has a description field, and the model automatically matches user intent to skill descriptions. You do not have to remember a skill’s command name; the description itself is the resolver.

This implies an important design principle: resolver quality depends on the quality of the description. If a skill description is too vague, such as “handle data-related tasks,” the model may trigger it when it should not or fail to trigger it when it should. A description is effectively a resolver’s routing rule and must be precise enough to route correctly.

This is exactly the same issue as a tool description in MCP Server development. An MCP tool’s description also acts as a resolver: the model decides which tool to call and when from the description. Writing a good tool description and writing a good skill description are the same underlying capability.

3.4 Practical Recommendation: Design AGENTS.md as an Index and Router

As projects multiply and patterns and gotchas accumulate, CLAUDE.md or AGENTS.md faces pressure to grow. Prepare early by designing it as an index and routing system rather than a knowledge encyclopedia.

For a project, AGENTS.md might contain only one line saying that the project’s development rules live in docs/AGENTS-project.md. That file then holds API quirks, special framework handling, testing strategy, and so on. The Agent reads it automatically when working on that project, but those details do not interfere with unrelated projects.

Put the right knowledge in the right place at the right time. At all other times, do not spend the limited attention budget.

4. The Most Important Design Decision: Latent vs. Deterministic

If you remember only one concept from this article, make it this one.

4.1 A Line That Must Be Drawn Clearly

Every step in an AI agent system belongs to exactly one of two categories:

Latent space is what an LLM does well: reading, understanding, judgment, synthesis, and pattern recognition. These tasks have no single correct answer and require intelligence.

Deterministic is what traditional code does well: the same input always produces the same output. Examples include SQL queries, compiled code, arithmetic, and combinatorial optimization.

Confusing the two is the most common error in agent design.

A vivid example: an LLM can arrange dinner seating for eight people by considering personalities and social relationships—a small judgment problem suitable for latent space. Ask it to seat 800 people, and it produces a plausible-looking but entirely wrong arrangement because seating 800 people is fundamentally a combinatorial-optimization problem, which is deterministic work forced into latent space.

4.2 Errors in Both Directions

Error 1: putting deterministic work in latent space. For example, ask an LLM to calculate the number of days between two dates. There is one correct answer, which one line of code can compute exactly. The LLM may be right, wrong, or off by one because it is not really “calculating”; it is guessing a plausible number.

Error 2: hard-coding latent work as deterministic logic. For example, use if-else statements and keyword matching to determine whether a user’s sentence is a complaint or a suggestion. This is fragile and has poor coverage because natural language can express an idea in infinitely many ways. The task inherently requires semantic understanding.

The worst systems contain both errors everywhere: they ask LLMs to perform arithmetic and exact retrieval, which LLMs do poorly, while using hard-coded rules for semantic understanding and content synthesis, which LLMs do well. The best systems are ruthless about this boundary.

“Ruthless” means allowing no ambiguity or expediency in classification. Developers are naturally tempted to take shortcuts. When already talking to an LLM, asking it to calculate a number or convert a format feels convenient. Accumulated shortcuts become the source of system unreliability. Good engineers resist this temptation just as they insist that business logic does not belong in a Controller in a Java project. Ruthlessness is not stubbornness; it is engineering discipline.

4.3 Drawing the Boundary in Practice

Consider a memory-association notification feature in a Telegram bot. Every step must be assigned deliberately to one side.

Step 1: retrieve historical memories semantically similar to a new memory from Mem0. → Deterministic. Call an API, pass a query, and receive top-K results. This is an API call with definite inputs and outputs and does not need LLM judgment.

Step 2: decide whether the retrieved historical memories have a meaningful relationship with the new memory. → Latent space. A “meaningful relationship” requires semantic judgment. Two memories may use completely different wording while describing two aspects of the same concept.

Step 3: decide whether to send a notification from the number of related memories and their relevance scores. → Deterministic. “Notify if there are at least two related memories and average relevance exceeds 0.7” requires only an if statement. An LLM may answer differently on different runs and make the user experience unpredictable.

5. Diarization: The LLM’s Most Irreplaceable Value

Once the latent-deterministic line is clear, a natural question follows: what is the most distinctive and irreplaceable ability of latent space? The answer is Diarization, used here to mean synthesis and distillation.

5.1 What Is Diarization?

The term originally comes from speech processing—speaker diarization—but in AI agent architecture it takes on a new meaning: the model reads a large amount of scattered information and distills it into one page of structured judgment, an analytical brief condensed from dozens or hundreds of documents.

The central claim is: no SQL query can produce this, and no RAG pipeline can produce this. The model must genuinely read, hold conflicting information simultaneously, notice what changed and when, and synthesize structured insight. It is the difference between a database query and an analyst’s brief.

5.2 Why RAG Cannot Perform Diarization

A common misconception needs clarification. Many people assume RAG can perform diarization because RAG also retrieves information from multiple documents and generates an answer. But the two are fundamentally different.

RAG centers on retrieval: given a query, find the most relevant document chunks and have the model answer from them. It solves the question, “Which information is relevant to my query?”

Diarization centers on synthetic judgment. It does not retrieve for one query. It reads all information about an entity—a person, project, or trend—then discovers contradictions, trajectories of change, and implicit patterns before producing an insightful analysis. It solves the question, “What does this information mean when considered together?”

A real example: a founder says in an application that she is building “Datadog for AI agents,” an observability tool, but 80% of her GitHub commits are in the billing module. This means she is actually building a FinOps tool wrapped in the language of observability.

Discovering the gap requires reading three different sources together—GitHub commit history, the application, and notes from adviser conversations—and cross-comparing them mentally. Vector search can find documents “related to FinOps,” but it cannot discover that what someone says differs from what they do. That is diarization’s exclusive territory.

5.3 Diarization Use Cases

The pattern transfers to many settings: user-behavior analysis, discovering the gap between what users say and do; competitive intelligence, synthesizing several sources to infer a competitor’s real strategy; code review, reading an entire pull request and making an architectural judgment rather than checking it line by line; and knowledge management, synthesizing fragmented user memories into a structured cognitive profile.

The central pattern is always the same: retrieve (deterministic) → read all material (latent) → synthesize judgment (latent) → emit structured insight (deterministic). Retrieval and output are deterministic; reading and judgment belong in latent space. The boundary remains clear.

6. A System That Learns: From One-Off Work to Permanent Improvement

The five preceding ideas—Spec, three-layer architecture, Resolver, Latent vs. Deterministic, and Diarization—are static design principles. Combining them into a self-improving loop is what changes the system qualitatively.

6.1 Designing the Learning Loop

Consider an event-matching system that intelligently groups 6,000 attendees by industry affinity, cross-industry serendipity, real-time pairing, and other factors.

After an event, an /improve skill reads the NPS survey and performs diarization. It does not analyze negative reviews, which are often isolated extremes; it specifically examines “it was okay” feedback, the places where the system almost succeeded but was not quite right. It extracts patterns, proposes new rules, and writes them back into the matching skill file:

When an attendee says "AI infrastructure"
    but 80%+ of the startup's code is billing-related:
    → Classify it as FinTech, not AI infrastructure.

When two attendees in the same group already know each other:
    → Penalize familiarity. Prefer novel introductions.

The rules take effect automatically on the next run. The Skill has rewritten itself.

At the first event, 12% of feedback was “okay.” At the next, it was 4%. The system improved without anyone rewriting code.

6.2 The Loop’s Structure

The learning loop is: run → collect feedback → analyze what “almost worked” → distill new rules → write them back to the skill → use them automatically next time.

The latent-deterministic boundary remains clear inside the loop. Analyzing NPS feedback, extracting patterns, and proposing rules belong in latent space. Writing the rules to a file and loading them on the next run are deterministic. The architectural principle stays consistent at every layer.

6.3 “If I Have to Ask You for the Same Thing Twice, You Have Failed”

This instruction expresses an architectural discipline precisely:

You are not allowed to perform one-off work. If I ask you to do something that will need to be done again, you must first complete three to ten samples manually and show me the results. If I approve, turn it into a skill file. If it should run automatically, attach it to a scheduled job.

This is not a prompt-engineering trick but a system-design philosophy. Every skill you write is a permanent upgrade to the system. It never degrades or forgets and runs at 3 a.m. while you sleep. When a next-generation model arrives, every skill immediately improves: judgment in latent-space steps improves automatically while deterministic steps remain perfectly reliable.

The practical implication is that if you repeatedly give an AI coding agent similar instructions—check that code style follows the rules, update the CHANGELOG, or summarize failure causes after tests—you should turn the instruction into a skill file or a permanent rule in AGENTS.md. Each such conversion permanently upgrades the system.

7. A Complete Mental Model

Putting the concepts together, the design model for an excellent AI agent system is:

Step 1: define intent with a Spec. Before code, state what you want, what you do not want, and which constraints apply. The Spec is the contract between you and AI and the system’s source of truth.

Step 2: draw the Latent-versus-Deterministic line. Inspect every system step and classify it ruthlessly. Give judgment to the LLM and precision to code. Allow no ambiguity.

Step 3: build the three-layer architecture. Push intelligence into skills—judgment and domain knowledge encoded in natural language. Push execution into deterministic tools—reliable API calls, database queries, and algorithms. Keep the harness thin and limited to scheduling.

Step 4: manage knowledge loading with a Resolver. Do not put all knowledge in one file. Design an index and routing system that puts the right knowledge in the right place at the right time.

Step 5: use Diarization for the LLM’s unique value. Where synthetic judgment is required, have the model read all material, hold contradictions, discover change, and produce insight. This ability is irreplaceable; do not try to substitute SQL or RAG.

Step 6: build a learning loop. Run → collect feedback → analyze what almost worked → distill new rules → write them back to the skill. Let the system improve itself and make every skill a permanent upgrade.

These six steps are not a linear pipeline but a continuously operating design framework. For every design decision, return to it and ask: is the Spec clear? Is this step on the correct side of the boundary? Is knowledge loaded at the right time? Can this capability become a skill?

Conclusion: The Engineer’s Core Capability in the AI Era

Return to the opening question: what kind of engineer is most valuable in the AI era?

Not the fastest coder—AI has already accelerated coding tenfold—but the engineer who can write a clear Spec to define intent, draw the latent-deterministic boundary for architectural decisions, design systems where AI intelligence and code determinism each do the right job, and build self-improving learning loops.

These capabilities have one thing in common. They are not code itself, but judgment about code: what should be written, what should not, what belongs with AI, what belongs with code, what should be codified into a rule, and what should remain flexible.

If you have traditional backend experience, these abilities are familiar. The discipline of keeping business logic out of a Controller in Java and keeping deterministic work out of latent space in an AI agent system are the same engineering instinct: separation of concerns, layered design, and doing the right work at the right level of abstraction. Your system now has one new component—the LLM—and you must learn to define its responsibility precisely.

Build once. Run forever. The system compounds.

References

• Addy Osmani, “How to write a good spec for AI agents”, January 2026
• Addy Osmani, “My LLM coding workflow going into 2026”, December 2025
• GitHub Blog, “Spec-driven development with AI”, September 2025
• GitHub Blog, “How to write a great agents.md: Lessons from over 2,500 repositories”, November 2025
• Martin Fowler (Birgitta Böckeler), “Understanding Spec-Driven-Development: Kiro, spec-kit, and Tessl”, 2025
• Kiro Documentation, “Specs”, 2026
• Andrew Ng, “Building at speed: The Product Management Bottleneck”, 2025