# Agent Tracing and Evaluation Setup

> Capture structured traces of every LLM and tool call and run a small regression eval suite on each change.

Category: AI Agent Workflows  
Tags: observability, tracing, evals, opentelemetry  
Setup time (estimate): 1-2 hours  
Estimated cost: Free tier of a tracing tool or self-hosted OpenTelemetry; storage grows with trace volume.  
Last verified: not yet independently verified

## What to log per span

```yaml
span:
  trace_id: uuid
  kind: llm | tool | retrieval
  name: model or tool name
  input: redacted prompt/args
  output: redacted response/result
  tokens: {in: int, out: int}
  latency_ms: int
  error: string|null
  prompt_version: git sha or tag
```

## Steps

1. Wrap your LLM client and tool dispatcher to emit spans (OpenTelemetry GenAI semantic conventions, or a tool such as Langfuse or Arize Phoenix, both of which can be self-hosted).
2. Redact secrets and personal data before export.
3. Build a golden set of 20-100 real tasks with expected outcomes.
4. On every prompt/model change, run the set and compare pass rate, cost, and latency against the previous version.
5. Promote failing production traces into the golden set.

## Failure modes

- Logging full prompts can store PII or secrets; redact before export.
- Eval sets that never change stop catching regressions; add real failures to them.
- High-volume tracing adds latency and cost if exported synchronously.