Designing Observability and Evaluation for RAG Systems

Preface

A RAG system returning HTTP 200 does not mean it has given the correct answer; Just because one answer seems reasonable doesn’t mean there are no issues with retrieval, citations, or permission control.

Monitoring for general applications usually focuses on latency, error rates, and resource usage. The RAG system also needs to answer another set of questions:

  • Which documents and chunks were used in this response?
  • What do Dense Retrieval, Sparse Retrieval, Fusion, and reranking each return?
  • Correct evidence was not recalled, or was it placed at the bottom after recall?
  • Is the model faithful to the evidence, and do the citations truly support the conclusion?
  • Does the system improve or deteriorate as a whole after prompt, model, or retrieval parameters change?

These issues cannot be solved by logs alone, nor by a single set of offline scores. A more complete design requires connecting observability and evaluation: the former records what the system actually does, while the latter judges whether it is well done.

If you need to first understand Trace, Metric, Log, Collector, and Context Propagation, you can read first: Getting Started with OpenTelemetry: A Popular Guide for AI Application and AI Agent Developers. This article will not repeat the full components of OpenTelemetry, but will focus on how they serve RAG’s quality closed loop.

1. Observability and Evaluation Answer Different Questions

observability and evaluation are closely related, but they should not be confused.

Dimensionobservabilityevaluation
Core issuesWhat actually happened to the systemIs the system well done?
Main targetsTrace、Span、Metric、LogDataset、Expected Output、Score、Experiment
Typical contentCall path, delay, token, error, candidate documentRecall, nDCG, correctness, faithfulness, citation quality
Time frameOnline requests and operational statusOffline regression and online quality assessment
Main usesIdentify faults and performance bottlenecksCompare plans and prevent quality regression

For example, a trace can illustrate:

retrieve.dense 返回了 20 个候选
retrieve.sparse 返回了 20 个候选
reranker.rank 耗时 180ms
llm.generate 使用了 3,200 个输入 Token

But it cannot be explained in isolation:

应该出现的文档是否真的被召回
排在前面的 Chunk 是否与问题相关
生成结果是否忠实于证据
引用是否支持对应结论

Therefore, Trace is an important data foundation for evaluation, but not the quality conclusion itself.

2. Decompose a RAG Request into Diagnosable Stages

When designing RAG Trace, do not create a Span for every small function. SPAN should have independent diagnostic value and allow for comparable delay or quality stages.

A typical link can be designed as:

Trace: rag.answer
├── request.validate
├── query.prepare
├── query.embed
├── retrieve
│   ├── retrieve.dense
│   ├── retrieve.sparse
│   └── retrieve.fuse
├── reranker.rank
├── context.build
├── llm.generate
├── citation.validate
└── response.finalize

If the system includes permission filtering, query rewriting, or multiple rounds of agents, it can also add:

query.rewrite
acl.filter
agent.plan
tool.call
guardrail.check

It is recommended to have a user request correspond to a root trace and link the API, queue, and Worker together through context propagation. If an asynchronous task does not pass a context, the entry request and background processing become two unrelated records.

What to Record at Each Stage

Structured attributes suitable for recording include:

service.name
service.version
deployment.environment.name
app.release.git_sha
app.prompt.version
app.knowledge_base.version
app.retrieval.config_hash

gen_ai.provider.name
gen_ai.request.model
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens

app.rag.top_k
app.rag.candidate_count
app.rag.document_ids
app.rag.chunk_ids
app.rag.reranker_model

Here, the app.* is a business-custom namespace, not an official OpenTelemetry field. Custom attributes should remain stable and clear, avoiding disguising them as official semantic conventions.

Version information is especially important. Without prompts, knowledge bases, embedding models, retrieval configurations, and Git SHA, a failed trace might only tell you “it went wrong,” but not “which version started to go wrong.”

3. Understand RAG Quality Through Four Metric Layers

RAG metrics are best layered by run, retrieval, generation, and business results. Just looking at the final answer score can obscure which layer the problem actually happened.

3.1 Runtime Metrics

Operational indicators reflect whether the system is stable, fast, and durable:

P50 / P95 / P99 Latency
Error Rate
Timeout Rate
Token Usage
Cost per Query
Empty Retrieval Rate
Index Freshness
Ingestion Failure Rate

These metrics are suitable for entering Metric and Dashboard, used for trending, alerting, and capacity assessment. trace_id. High base or sensitive content such as user IDs and question bodies should not be used as Metric Labels.

3.2 Retrieval Metrics

retrievalevaluation answered, “Did the evidence get it right?”:

Recall@K
Precision@K
MRR
nDCG@K
Document Hit Rate
ACL Leakage Rate

If the correct document does not enter the candidate set at all, check chunking, embedding, query rewriting, and recall channels; If it makes it into the candidate set but does not rank at the top, you should check fusion and reranking. Storing candidate results in stages is necessary to distinguish between these two types of issues.

3.3 Generation Metrics

Generate evaluation answer “Did the model use evidence correctly?”:

Answer Correctness
Groundedness / Faithfulness
Answer Relevance
Citation Correctness
Citation Completeness
Refusal Accuracy

These metrics may come from deterministic rules, manual annotation, or calibrated LLM-as-a-Judge. When subjective judgment is involved, the model score should not be treated as absolute truth; Manual samples should first be used to examine scoring criteria, consistency, and bias.

3.4 Business Outcomes

Whether the system ultimately has value depends on the actual task results:

用户是否解决问题
是否继续追问或改写问题
点赞、点踩和人工纠错
任务完成率
转人工率
高风险错误数量

Business outcomes are usually closer to true goals than a single offline metric, but feedback may be delayed. Subsequent feedback should be linked back to the current runtime version and evidence chain through stable trace ID, session ID, or business task ID.

4. Evaluation Datasets Need an Independently Versioned Source of Truth

observability platform can provide Dataset, Experiment, and Score interfaces, but evaluation data should not exist only on a single webpage.

A prudent approach is to use JSONL, JSON, or other censorable files in the repository as version source of truth:

{
  "id": "refund-policy-001",
  "question": "退款申请需要哪些材料？",
  "expected_behavior": "列出必要材料并给出政策出处",
  "expected_sources": ["refund-policy"],
  "tags": ["policy", "citation"]
}

Each evaluation run must record at least the following:

dataset_hash
git_sha
prompt_version
knowledge_base_version
embedding_provider_and_model
retrieval_parameters
reranker_model
generation_model
evaluation_code_version

dataset_hash Used to confirm whether the test content has changed; git_sha and configure fingerprints to confirm the system version under test. Only by freezing input data, system configuration, execution results, and scoring rules together can two Experiments truly be comparable.

Datasets within the platform are better suited as entry points for collaboration, annotation, and experimentation. If synchronization fails, the local dataset and runtime products should still be retained and not lost due to temporary inavailability evaluation control plane history.

5. Build a Feedback Loop Between Online and Offline Systems

Offline evaluation and online observation should not be two separate systems. Mature processes are usually:

生产 Trace
  ↓
规则、用户反馈、人工审核或 LLM Judge 评分
  ↓
发现失败样本与边界案例
  ↓
加入版本化 Evaluation Dataset
  ↓
修改 Prompt、模型、Chunking、Retriever 或 Reranker
  ↓
运行离线 Experiment
  ↓
比较基线并执行 CI Regression Gate
  ↓
灰度发布并继续观察

This closed loop can avoid two common problems:

  • Offline datasets remain unupdated for a long time, disconnected from real user issues;
  • Online, only a large amount of trace is accumulated, but failures are not converted into repeatable tests.

For RAG, it is recommended to first use a retrieval metric with strong deterministic closed-loop determinism, then expand to generation quality and LLM-as-a-Judge. retrieval stage is usually easier to establish a clear Ground Truth and more easily pinpoint the causal relationship between changes and indicator changes.

For a more complete RAG layered architecture and evaluation metrics, you can refer to: Enterprise-level RAG system setup guide.

6. Where the Observability and Evaluation Control Plane Fits

One common architecture is to let business applications generate structured telemetry and evaluation products, which are then stored, displayed, and analyzed by different backends:

RAG Application
├── HTTP / Worker / Evaluation Runner
├── OpenTelemetry Instrumentation
└── Versioned Evaluation Dataset
            │
            ├── Trace / Metric / Log
            │        ↓
            │   Collector（可选）
            │        ↓
            │   Observability Backend
            │
            └── Dataset / Experiment / Score
                     ↓
                Evaluation Control Plane

Here, the Observability Backend and Evaluation Control Plane can be handled by a single platform or split apart. When choosing, focus on capability boundaries rather than binding products first:

  • Whether parent-child traces at each RAG stage can be shown;
  • Whether token, cost, and model information can be recorded;
  • Whether Dataset, Experiment, Score, and manual annotation are supported;
  • Interoperability with OpenTelemetry;
  • Whether data residency, permissions, backup, and upgrade requirements are met;
  • Will backend inavailability affect business requests?

Langfuse, Phoenix, and others belong to this layer of implementation options. Take Langfuse as an example: it can place Trace, Dataset, Experiment, and Score in the same control plane; The current Python SDK is based on OpenTelemetry and can be shared context with other OTel Instrumentations.

If self-managed is adopted, product selection becomes an operations and maintenance choice. Taking the low-scale Docker Compose deployment of Langfuse v3 as an example, it requires maintaining Web, Worker, PostgreSQL, ClickHouse, Redis/Valkey, and object storage. Docker Compose is suitable for testing or low-scale environments, but it does not offer full high availability, horizontal scaling, or automatic backup capabilities.

This shows that “keeping data in your own environment” does not mean “no cost,” but rather converts hosting costs into capacity, backup, recovery, upgrades, and security responsibilities.

7. Avoid Duplicate Reporting and Incomplete Traces

An application may simultaneously access frame callbacks, model gateway callbacks, OpenTelemetry automatic staking stakes, and platform SDKs. If they all create Generation for the same model call, it is easy to generate duplicate traces, duplicate tokens, and recurring costs.

Before connecting, clarify:

谁创建根 Trace
谁创建 Retrieval Span
谁创建 Generation Observation
谁负责自动插桩 HTTP、数据库和队列
哪些 Span 发送到 AI 可观测后端
哪些信号发送到基础设施可观测后端

If the AI control plane and infrastructure backend share a TracerProvider, data can be distributed through different Span Processors or Exporters, but clear filtering rules should be made to avoid sending all low-value infrastructure Span to the AI platform.

If using providers isolated from each other, verify that the parent-child relationship and context are still intact. Incorrect provider boundaries may lead to isolated spans, resulting in only scattered model calls and missing the full RAG request.

8. Do Not Collect Body Content by Default; Enforce Privacy at the Application Layer

RAG Trace easily includes user questions, private documentation, prompts, evidence text, and model responses. For easier debugging, all content is recorded by default, quickly generating new copies of sensitive data.

The safer default policy is:

capture_content = false

By default, only structured information from the allowed list is sent, for example:

模型和提供商
Token 与时延
候选数量
Document ID / Chunk ID
检索和重排参数
状态、错误类型和业务结果

The following content should be disabled by default and only enabled after explicit authorization and retention policy review:

用户问题正文
私有文档与证据正文
完整 Prompt
模型输入与输出
认证信息和个人信息

The first layer of desensitization should occur on the application or SDK side, ensuring sensitive content is removed before sending. Collector or platform-side secondary filtering can serve as a safety net, but cannot replace client boundaries; Once data leaves the application, the complexity of subsequent deletion and access control increases significantly.

9. Observability Must Be a Gracefully Degradable Side Path

Telemetry and evaluationcontrol plane are used to understand business and should not become a hard dependency on the business.

The following principles are recommended:

  • Batch processing and asynchronous export are used to avoid waiting for telemetry backends in master requests;
  • Exporter logs network errors but does not change normal business returns;
  • Short-lifecycle processes explicitly flush or shutdown before exiting;
  • Local evaluation results are first placed on the disk, then asynchronously synchronized to the remote control plane;
  • Monitor queue capacity, drop volume, retry count, and disk usage;
  • Credential configuration errors should be exposed as early as possible when telemetry is enabled, and data should not be lost silently for extended periods.

“Business continues when the backend is unavailable” and “rapid failure due to configuration errors” are not mutually exclusive: the former handles external runtime faults, while the latter prevents the system from running for extended periods when it appears enabled but actually has no data.

You don’t have to build a full platform on day one. Construction can be carried out step by step as follows:

Phase 1: Unify Logs and Request Identity

  • Assigning a stable Request ID to requests;
  • Use structured JSON logs;
  • Record models, tokens, latency, errors, and behavioral outcomes;
  • Do not collect sensitive body text.

Phase 2: Establish Core Traces

  • A RAG request corresponds to one root Trace;
  • Create manual Span for Retrieval, Reranking, and Generation;
  • Integrate HTTP, queues, and Worker contexts;
  • Write versions of Git SHA, prompts, knowledge bases, and retrieval configurations.

Phase 3: Add Metrics and a Control Plane

  • Aggregation latency, error rate, tokens, cost, and retrieval rate;
  • Connecting to Collector or observability backend;
  • Validation sampling, filtering, retrying, and fault degradation;
  • Establish backup, capacity, and upgrade responsibilities.

Phase 4: Establish Repeatable Evaluation

  • Preparing the versioned Golden Dataset;
  • First, complete retrieval indicators such as Recall, MRR, and nDCG;
  • Preserve sample-level results and aggregated metrics;
  • Record Dataset Hash, Git SHA, and runtime configuration.

Phase 5: Close the Continuous Improvement Loop

  • Add the online failure sample to the Dataset;
  • Introducing manual feedback and calibrated LLM Judges;
  • Run experiments on prompts, models, and retrieval schemes;
  • Set validated regression thresholds in CI.

11. Design Checklist

Before going live, you can check item by item:

Trace

  • Can a request see the complete link from query processing to final response?
  • Do APIs, queues, and workers share the same context?
  • Can you position the failed answer into specific retrieval candidate and running version?

Evaluation

  • Are evaluationretrieval, generated, and business outcomes all at the same time?
  • Is the dataset versionable, reviewable, and reusable?
  • Does the Experiment record Dataset Hash, Git SHA, models, and retrieval configurations?

Privacy

  • Is it by default to turn off collection of questions, evidence, prompts, and answers?
  • Is sensitive content filtered before leaving the app?
  • Are there data retention, deletion, and access control policies?

Reliability

  • When the telemetry backend is unavailable, can the business still respond normally?
  • Are local evaluation results stored independently?
  • Is there someone responsible for queues, abandonment, disks, backups, and recovery?

Conclusion

RAG observability not just adds a trace to every model call, and evaluation doesn’t just run LLM-as-a-Judge once.

A more complete system should form relationships such as:

Trace 记录一次请求经历了什么
Metric 展示系统整体表现如何
Log 保存具体事件和错误细节
Dataset 固化需要重复验证的问题
Score 表达质量判断
Experiment 比较不同版本的结果
线上失败继续反哺离线回归

The truly valuable goal is not to “collect more data,” but to enable a failure to be located, explained, added to the test set, and reliably reproduced before the next release.

References