MODEL EVALUATION / Datadog Labs
Datadog Trace RCA
Root-causes production AI failures from LLM traces through phased span analysis, then recommends prompt, tool, routing or retrieval changes quoted from the trace evidence.
“Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval judge verdicts, runtime errors, or structural anomalies depending on what signals are present. Walks the span tree from symptom to root cause. Use when user says "what's wrong with my app", "why is my eval failing", "analyze errors", "root cause analysis", "diagnose failures", or wants to understand production failure patterns.”
01 / THE REASONING
Why this made the selection.
- Structures the analysis as explicit phases from input resolution through open and axial coding to recommendations, instead of offering generic debugging advice.
- Bundles its own eval harness with a judge rubric, example config, and executor, so the method is checkable rather than asserted.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- agent-observability/agent-observability-trace-rca/SKILL.md
- agent-observability/agent-observability-trace-rca/evals/prompts/judge_rubric.txt
- agent-observability/agent-observability-trace-rca/evals/example_config.json
- LICENSE
Our findings
- The workflow separates eval-signal, error-signal, and generic analysis modes with distinct signal-summary formats.
- Span-content and span-search patterns are specified per tool with parallelization rules for batched fetches.
- Phase 7 takes no action automatically: it asks whether to save the report, apply fixes, investigate further, export a Datadog notebook or re-run on a longer window. Evaluator suites are left to a separate /eval-bootstrap step.
Not established by this review
- No MCP tools were called and no production traces were analyzed.
- The bundled evaluator and judge rubric were read but not executed.
- Not executed: it was not installed or run.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Add a skill to your workflow.
DOCUMENTED COMMAND
npx skills add datadog-labs/agent-skills/agent-observability/agent-observability-trace-rca --skill agent-observability-trace-rcaRun in your terminal to choose your AI tools, project or global scope, and installation method. This short command follows the current upstream version; the review covers the revision shown above.
Optional: install the reviewed revision
Keeps the original source and installer version where recorded, while leaving your AI tools and installation scope for you to choose.
npx skills add https://github.com/datadog-labs/agent-skills/tree/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca --skill agent-observability-trace-rca- Install the complete skill directory with its evals; a copied SKILL.md alone is incomplete.
- Choose your AI tools and project or global scope in the installer, then review its permissions before using the skill.
Before you start
- Node.js/npm for this installer
- An agent that supports the Agent Skills format
- Datadog LLMO MCP toolset with API credentials and populated LLM Observability data
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca/SKILL.md
Supports: summary, upstreamDescription, whySelected, bestFor, limitations, review, compatibility, access
https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca/evals/prompts/judge_rubric.txt
Supports: review, whySelected
- Upstream license ↗Checked
https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/LICENSE
Supports: license
https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/README.md
Supports: install, compatibility, access
https://github.com/vercel-labs/skills/blob/48dc9e8eb8aa19040e91cb90ba0b642c9cbad57e/README.md
Supports: install
https://github.com/vercel-labs/skills/blob/48dc9e8eb8aa19040e91cb90ba0b642c9cbad57e/src/git.ts
Supports: install
https://raw.githubusercontent.com/datadog-labs/agent-skills/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca/SKILL.md
Supports: upstreamDescription, review.notTested, review.reviewedAt, limitations