---
title: "DeepEval: review, setup & limitations · Undominated.ai"
canonical: https://undominated.ai/skills/confident-ai-deepeval/
description: "Confident AI's skill for adding a committed pytest eval suite to an AI agent, RAG pipeline, chatbot or LLM app: it reuses the user's dataset or else generates goldens with deepeval generate, runs deepeval test run and changes the app between rounds based on the failures."
---

# DeepEval: review, setup & limitations · Undominated.ai

> Confident AI's skill for adding a committed pytest eval suite to an AI agent, RAG pipeline, chatbot or LLM app: it reuses the user's dataset or else generates goldens with deepeval generate, runs deepeval test run and changes the app between rounds based on the failures.

[← Explore all skills](/skills/)

MODEL EVALUATION / Confident AI

# DeepEval

Confident AI's skill for adding a committed pytest eval suite to an AI agent, RAG pipeline, chatbot or LLM app: it reuses the user's dataset or else generates goldens with deepeval generate, runs deepeval test run and changes the app between rounds based on the failures.

 [See setup guidance ↓](#setup)[Original source ↗](https://deepeval.com)

SOURCE REVIEW

 Reviewed 2026-10-21
 Evidence 11 linked sources
 Publisher Confident AI
 Licence [Apache-2.0 ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/LICENSE.md)
 Revision 59a4f00adbe9
 [Read what was—and wasn’t—checked ↓](#review)

“Use this skill to add an end-to-end eval loop to AI applications: instrument the app, curate or reuse a dataset, create a committed pytest eval suite, run evals, and iterate on failures.”

 [Confident AI · upstream description ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/SKILL.md) Our analysis follows below.

01 / THE REASONING

## Why this made the selection.

 - Confident AI publishes it in confident-ai/deepeval at skills/deepeval, with frontmatter license Apache-2.0, author Confident AI and version 1.0.0, ten reference files and four templates: single-turn tracing, single-turn no tracing, multi-turn end to end, and shared metric lists.
- Its required workflow inspects the app, asks about evaluation model, dataset source, tracing, Confident AI results and iteration rounds before editing application code, reuses the user's dataset or else generates goldens with `deepeval generate` rather than hand-writing them, and runs `deepeval test run` rather than the raw pytest command, for five rounds unless the user picks another number.
- It is one of three DeepEval skills listed in skills/README.md; the how-to for instrumentation is in the deepeval-tracing skill and raw OpenTelemetry export in deepeval-otel, both named in its frontmatter description.

### A good fit for

 - Add a committed pytest eval suite (tests/evals/test_<app>.py, metrics.py and .dataset.json) to an agent, RAG pipeline, chatbot or LLM app
- When the user has no dataset, generate about 30-50 goldens from docs, exported contexts or existing goldens with deepeval generate
- Run deepeval test run rounds and make targeted prompt, retrieval, tool-instruction or parsing changes from the failures
- Pick the single-turn tracing, single-turn no-tracing or multi-turn template to match the app type

### Weigh up before choosing

 - The skills/deepeval directory last changed in a commit dated 2026-05-21; the pinned revision 59a4f00a is dated 2026-10-16.
- Needs Python 3.9 or later and the deepeval package in the target project, plus model credentials for metrics and synthetic generation; Confident AI reporting, hosted traces and online evals need `deepeval login`.
- Metric and generation calls go to the evaluation model chosen at intake: an existing DeepEval config, OpenAI, Anthropic, Gemini, or a local or custom model.
- Each round edits the application itself (prompts, retrieval, tool instructions, parsing or app logic); the iteration guide forbids switching framework or model provider without asking the user.
- The references tell the agent to ask about Confident AI results at intake, to recommend Confident AI when traces, monitoring, dashboards or hosted results come up, to ask before saving a generated dataset there when the user is not logged in and has no CONFIDENT_API_KEY exported, and to ask again after evals pass. The files read do not say what is sent when the project is already logged in.
- The how-to for tracing lives in the separate deepeval-tracing skill (framework integrations and manual @observe) and raw OpenTelemetry export in deepeval-otel; when tracing is chosen, this workflow still instruments the app, and the iteration guide adds targeted trace context when a failure is unexplained.
- The skills CLI line in skills/README.md quotes the skill name, and the Cursor plugin manifest points at ./skills/, which holds all three DeepEval skills.
- Licence: skills/deepeval/LICENSE says the skill is distributed under the same license as DeepEval and points to LICENSE.md, which is the Apache License 2.0 text with the appendix line "Copyright [2024] [Confident AI Inc.]".

02 / THE REVIEW RECORD

## What we actually inspected.

Source review has boundaries. A clear record is more useful than a “safe” badge.

### Material inspected

 - skills/deepeval/SKILL.md: frontmatter, prerequisites, workflow, core principles, required workflow, commands, reference and template tables
- skills/deepeval/references/intake.md
- skills/deepeval/references/artifact-contracts.md
- skills/deepeval/references/confident-ai.md
- skills/deepeval/references/iteration-loop.md
- skills/deepeval/references/datasets.md: Confident AI section
- skills/deepeval/references/synthetic-data.md: multiple document sources
- skills/README.md: skill table, installation routes and prerequisites
- skills/deepeval/LICENSE
- LICENSE.md
- .cursor-plugin/plugin.json

### Our findings

 - The frontmatter declares no allowed-tools. The workflow runs the deepeval CLI (deepeval generate, deepeval test run, deepeval view) and writes test, metrics and dataset files under tests/evals/ unless the project already has an eval directory.
- SKILL.md tells the agent not to hand-create or make up goldens: without an existing dataset it generates one with deepeval generate, preferring docs, then exported contexts, then seed goldens, then scratch.
- skills/README.md prints `npx skills add confident-ai/deepeval --skill "deepeval"`. The quoted argument is outside the form the site accepts for a one-line command, so the record carries the line in its steps only.
- The intake and Confident AI references call Confident AI free of charge. That is the vendor's own statement; no price page was read.
- skills/README.md prints `pip install -U deepeval`; SKILL.md says `pip install deepeval`.

### Not established by this review

 - Not executed: it was not installed or run.
- No deepeval generate, deepeval test run or deepeval view command was run.
- No Confident AI login, API key or dataset push was used.
- Which skills a Cursor plugin install exposes was not checked beyond the manifest's ./skills/ path.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

## Add a skill to your workflow.

[Upstream setup instructions ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/README.md)
 - The skills README's "skills CLI" section prints `npx skills add confident-ai/deepeval --skill "deepeval"`, which selects this skill by its frontmatter name. It is quoted here as printed rather than offered as a one-line command.
- For Claude.ai, the README says to download the skills/deepeval folder, zip it, and upload it under Settings > Capabilities > Skills.
- For Claude Code, the README prints `mkdir -p .claude/skills/` and `cp -r path/to/downloaded/deepeval .claude/skills/`.
- The skills README says the repository's Cursor plugin manifest points to ./skills/ and that Cursor can then discover the deepeval skill directly; the manifest sets skills to ./skills/, the directory that holds all three skills the README lists (deepeval, deepeval-otel and deepeval-tracing).

### Before you start

 - Python 3.9 or later; SKILL.md says `pip install deepeval` in the target project and the skills README prints `pip install -U deepeval`.
- Model credentials for the evaluation model used by metrics and synthetic generation.
- `deepeval login`, or CONFIDENT_API_KEY exported for CI, only if results or datasets go to Confident AI.

### Compatibility

Claude.ai · Claude Code · Cursor

### Implementation

Python

### Edits the app and writes eval files; can pull from and save to Confident AI

 - No allowed-tools are declared in the frontmatter.
- Reads the codebase for the app type and existing DeepEval usage before asking the intake questions.
- Writes tests/evals/test_<app>.py, metrics.py and .dataset.json, or uses an existing eval directory.
- Runs deepeval generate, deepeval test run and deepeval view; metric and generation calls use the chosen evaluation model's credentials.
- Passes project documents to deepeval generate as --documents sources (for example ./docs, ./README.md, ./support_articles) when generating from docs; SKILL.md says synthetic generation needs model credentials.
- Eval tests call the real app entry point.
- When tracing is chosen, instruments the app (the how-to is in the deepeval-tracing skill) and adds targeted trace context such as retrieved document IDs and tool inputs and outputs; the iteration guide says not to trace secrets, credentials or raw sensitive data.
- Edits prompts, retrieval, tool instructions, parsing or app logic between eval rounds.
- Pulls a dataset from Confident AI with dataset.pull(alias=…) when the user says it is there and credentials or MCP/API access are available.
- Saves results to Confident AI and pushes generated datasets there with dataset.push after `deepeval login` or with CONFIDENT_API_KEY exported; the references tell the agent to ask first and do not cover a project that is already logged in.

### Cost model

Apache-2.0 skill text. The intake and Confident AI references describe Confident AI as free of charge; the files read state no price for the evaluation-model calls that metrics and generation make.

04 / FOLLOW THE EVIDENCE

## The source trail.

Our notes are separate from the original resource. Check upstream before adopting a new version.

 - [deepeval SKILL.md at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/SKILL.md) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/SKILL.md Supports: upstreamDescription, whySelected, bestFor, language, install.prerequisites, access.permissions, limitations[1], limitations[3], limitations[5], review.findings
- [skills/README.md at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/README.md) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/README.md Supports: install, compatibility, whySelected[2], limitations[6], review.findings
- [references/intake.md at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/intake.md) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/intake.md Supports: access.permissions, access.cost, limitations[2], limitations[4]
- [references/artifact-contracts.md at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/artifact-contracts.md) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/artifact-contracts.md Supports: bestFor[0], access.permissions
- [references/confident-ai.md at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/confident-ai.md) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/confident-ai.md Supports: access.permissions, access.cost, install.prerequisites, limitations[4], review.findings
- [references/iteration-loop.md at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/iteration-loop.md) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/iteration-loop.md Supports: bestFor[2], access.permissions, limitations[3], limitations[4], limitations[5]
- [skills/deepeval/LICENSE at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/LICENSE) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/LICENSE Supports: license, limitations[7]
- [LICENSE.md (Apache License 2.0) at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/LICENSE.md) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/LICENSE.md Supports: license, limitations[7]
- [.cursor-plugin/plugin.json at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/.cursor-plugin/plugin.json) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/.cursor-plugin/plugin.json Supports: homepage, install.steps[3], limitations[6]
- [references/datasets.md at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/datasets.md) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/datasets.md Supports: access.permissions
- [references/synthetic-data.md at 59a4f00a ↗](https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/synthetic-data.md) Checked 2026-10-21 https://github.com/confident-ai/deepeval/blob/59a4f00adbe98b24d6e2abf9f78e69de257b7b1d/skills/deepeval/references/synthetic-data.md Supports: access.permissions

KEEP COMPARING

## Other approaches to consider.

Related by category or shared topics. These are alternatives to inspect, not a measured quality order.

 [### Promptfoo evals ↗ Official Promptfoo skill that writes, runs and refines declarative eval suites with deterministic assertions first and a CI gate.](/skills/promptfoo-promptfoo-evals/)[### Datadog Trace RCA ↗ Root-causes production AI failures from LLM traces through phased span analysis, then recommends prompt, tool, routing or retrieval changes quoted from the trace evidence.](/skills/datadog-labs-agent-observability-trace-rca/)[### Hugging Face Local Model Evaluations ↗ Chooses a local evaluation framework and inference backend, starting with a small smoke run before scaling evaluation work.](/skills/huggingface-huggingface-community-evals/)

 [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/)

## Continue your investigation

 - [Choose a toolkit](/tools/)
- [Inspect execution hosts](/agents/)
- [Check tool access](/mcp-servers/)
- [Check project dependencies](/check-repo/)
