--- title: "Datadog Trace RCA: review, setup & limitations · Undominated.ai" canonical: https://undominated.ai/skills/datadog-labs-agent-observability-trace-rca/ description: "Root-causes production AI failures from LLM traces through phased span analysis, then recommends prompt, tool, routing or retrieval changes quoted from the trace evidence." --- # Datadog Trace RCA: review, setup & limitations · Undominated.ai > Root-causes production AI failures from LLM traces through phased span analysis, then recommends prompt, tool, routing or retrieval changes quoted from the trace evidence. [← Explore all skills](/skills/) MODEL EVALUATION / Datadog Labs # Datadog Trace RCA Root-causes production AI failures from LLM traces through phased span analysis, then recommends prompt, tool, routing or retrieval changes quoted from the trace evidence. [See setup guidance ↓](#setup)[Original source ↗](https://github.com/datadog-labs/agent-skills/tree/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca) SOURCE REVIEW Reviewed 2026-10-15 Evidence 7 linked sources Publisher Datadog Labs Licence [MIT ↗](https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/LICENSE) Revision 9f3a8b065be6 [Read what was—and wasn’t—checked ↓](#review) “Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval judge verdicts, runtime errors, or structural anomalies depending on what signals are present. Walks the span tree from symptom to root cause. Use when user says "what's wrong with my app", "why is my eval failing", "analyze errors", "root cause analysis", "diagnose failures", or wants to understand production failure patterns.” [Datadog Labs · upstream description ↗](https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca/SKILL.md) Our analysis follows below. 01 / THE REASONING ## Why this made the selection. - Structures the analysis as explicit phases from input resolution through open and axial coding to recommendations, instead of offering generic debugging advice. - Bundles its own eval harness with a judge rubric, example config, and executor, so the method is checkable rather than asserted. ### A good fit for - Explaining why an AI feature regressed using production traces - Turning a diagnosed failure pattern into a concrete prompt, tool or routing change, proposed as a diff before any file is edited ### Weigh up before choosing - Requires either the Datadog LLMO MCP toolset or the pup CLI with usable authentication and existing LLM Observability traces. - Report quality depends on trace volume and eval coverage, and no evaluator effectiveness benchmark was run for this review. 02 / THE REVIEW RECORD ## What we actually inspected. Source review has boundaries. A clear record is more useful than a “safe” badge. ### Material inspected - agent-observability/agent-observability-trace-rca/SKILL.md - agent-observability/agent-observability-trace-rca/evals/prompts/judge_rubric.txt - agent-observability/agent-observability-trace-rca/evals/example_config.json - LICENSE ### Our findings - The workflow separates eval-signal, error-signal, and generic analysis modes with distinct signal-summary formats. - Span-content and span-search patterns are specified per tool with parallelization rules for batched fetches. - Phase 7 takes no action automatically: it asks whether to save the report, apply fixes, investigate further, export a Datadog notebook or re-run on a longer window. Evaluator suites are left to a separate /eval-bootstrap step. ### Not established by this review - No MCP tools were called and no production traces were analyzed. - The bundled evaluator and judge rubric were read but not executed. - Not executed: it was not installed or run. The review applies to the material and revision named here. A newer upstream release can change its behavior. 03 / PUT IT TO WORK ## Add a skill to your workflow. [Upstream setup instructions ↗](https://github.com/vercel-labs/skills/blob/48dc9e8eb8aa19040e91cb90ba0b642c9cbad57e/README.md) DOCUMENTED COMMAND npx skills add datadog-labs/agent-skills/agent-observability/agent-observability-trace-rca --skill agent-observability-trace-rca Copy command ↗ Run in your terminal to choose your AI tools, project or global scope, and installation method. This short command follows the current upstream version; the review covers the revision shown above. Optional: install the reviewed revision Keeps the original source and installer version where recorded, while leaving your AI tools and installation scope for you to choose. npx skills add https://github.com/datadog-labs/agent-skills/tree/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca --skill agent-observability-trace-rca Copy reviewed-revision command ↗ - Install the complete skill directory with its evals; a copied SKILL.md alone is incomplete. - Choose your AI tools and project or global scope in the installer, then review its permissions before using the skill. ### Before you start - Node.js/npm for this installer - An agent that supports the Agent Skills format - Datadog LLMO MCP toolset with API credentials and populated LLM Observability data ### Compatibility Agent Skills-compatible coding agents · Datadog LLM Observability with the LLMO MCP toolset ### Datadog observability data via MCP - Query LLM traces, spans, evals, and session data - Generate evaluator code files when authorized - Trace queries and evaluator runs consume Datadog and agent resources ### Cost model The skill text is MIT-licensed; Datadog usage and agent calls have their own costs. 04 / FOLLOW THE EVIDENCE ## The source trail. Our notes are separate from the original resource. Check upstream before adopting a new version. - [Skill definition at reviewed revision ↗](https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca/SKILL.md) Checked 2026-10-13 https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca/SKILL.md Supports: summary, upstreamDescription, whySelected, bestFor, limitations, review, compatibility, access - [Reviewed helper: evals/prompts/judge_rubric.txt ↗](https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca/evals/prompts/judge_rubric.txt) Checked 2026-10-07 https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca/evals/prompts/judge_rubric.txt Supports: review, whySelected - [Upstream license ↗](https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/LICENSE) Checked 2026-10-07 https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/LICENSE Supports: license - [Publisher installation documentation ↗](https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/README.md) Checked 2026-10-07 https://github.com/datadog-labs/agent-skills/blob/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/README.md Supports: install, compatibility, access - [Skills installer source formats and selective installation ↗](https://github.com/vercel-labs/skills/blob/48dc9e8eb8aa19040e91cb90ba0b642c9cbad57e/README.md) Checked 2026-10-07 https://github.com/vercel-labs/skills/blob/48dc9e8eb8aa19040e91cb90ba0b642c9cbad57e/README.md Supports: install - [Installer implementation: full commit SHA fallback ↗](https://github.com/vercel-labs/skills/blob/48dc9e8eb8aa19040e91cb90ba0b642c9cbad57e/src/git.ts) Checked 2026-10-07 https://github.com/vercel-labs/skills/blob/48dc9e8eb8aa19040e91cb90ba0b642c9cbad57e/src/git.ts Supports: install - [2026-10-15 C2-W03 source re-review at the existing pin ↗](https://raw.githubusercontent.com/datadog-labs/agent-skills/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca/SKILL.md) Checked 2026-10-15 https://raw.githubusercontent.com/datadog-labs/agent-skills/9f3a8b065be670e70bb9b66999aae0b786a0bb7d/agent-observability/agent-observability-trace-rca/SKILL.md Supports: upstreamDescription, review.notTested, review.reviewedAt, limitations KEEP COMPARING ## Other approaches to consider. Related by category or shared topics. These are alternatives to inspect, not a measured quality order. [### Promptfoo evals ↗ Official Promptfoo skill that writes, runs and refines declarative eval suites with deterministic assertions first and a CI gate.](/skills/promptfoo-promptfoo-evals/)[### Agent Platform Eval Flywheel ↗ Evaluates and iterates on Google Cloud Agent Platform models and agents, with a confirmation card before every billed evaluation run.](/skills/google-agent-platform-eval-flywheel/)[### Arize Evaluator ↗ Creates and runs Arize LLM-as-judge and code evaluators through the ax CLI, with a rule against fabricating scores.](/skills/arize-ai-arize-evaluator/) [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/) ## Continue your investigation - [Choose a toolkit](/tools/) - [Inspect execution hosts](/agents/) - [Check tool access](/mcp-servers/) - [Check project dependencies](/check-repo/)