--- title: "Promptfoo evals: review, setup & limitations · Undominated.ai" canonical: https://undominated.ai/skills/promptfoo-promptfoo-evals/ description: "Official Promptfoo skill that writes, runs and refines declarative eval suites with deterministic assertions first and a CI gate." --- # Promptfoo evals: review, setup & limitations · Undominated.ai > Official Promptfoo skill that writes, runs and refines declarative eval suites with deterministic assertions first and a CI gate. [← Explore all skills](/skills/) MODEL EVALUATION / Promptfoo # Promptfoo evals Official Promptfoo skill that writes, runs and refines declarative eval suites with deterministic assertions first and a CI gate. [See setup guidance ↓](#setup)[Original source ↗](https://www.promptfoo.dev/docs/integrations/agent-skill) SOURCE REVIEW Reviewed 2026-10-15 Evidence 3 linked sources Publisher Promptfoo Licence [MIT ↗](https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/LICENSE) Revision 8570ed47327c [Read what was—and wasn’t—checked ↓](#review) “Write, run, and improve non-redteam Promptfoo eval suites for a configured target: test cases, assertions, rubrics, datasets, and CI gates. Use promptfoo-provider-setup first for a new or broken connection; use the redteam skills for adversarial scans.” [Promptfoo · upstream description ↗](https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/plugins/promptfoo/skills/promptfoo-evals/SKILL.md) Our analysis follows below. 01 / THE REASONING ## Why this made the selection. - Defines a non-redteam eval workflow with deterministic assertions before model-graded rubrics, inlined source evidence, calibrated known-good and known-bad controls, and an exit-code CI gate. - Declares explicit handoffs to the sibling provider-setup skill for new or broken connections and to the redteam skills for adversarial scans, instead of covering everything in one file. ### A good fit for - Write a declarative promptfooconfig suite with prompts, providers, variables, tests and assertions - Calibrate a grader with known-good and known-bad outputs before gating a change - Run validate and eval with no-cache and no-share flags and inspect the results ### Weigh up before choosing - Last commit 2026-10-15, the same day as the review; the bundle ships four skills and this candidate is the eval-authoring one only. - Requires a configured provider target with credentials; model and grader calls send data to the configured providers even with sharing disabled. - Mock graders verify wiring only and cannot replace a real quality judgment. - Adversarial scanning is out of scope and routes to the sibling redteam skills. - Licence evidence is the MIT grant in LICENSE with copyright to Promptfoo 2025, confirmed by MIT declarations in both plugin manifests. 02 / THE REVIEW RECORD ## What we actually inspected. Source review has boundaries. A clear record is more useful than a “safe” badge. ### Material inspected - plugins/promptfoo/skills/promptfoo-evals/SKILL.md - plugins/promptfoo/skills/promptfoo-evals/references/eval-patterns.md - site/docs/integrations/agent-skill.md - LICENSE ### Our findings - The skill routes provider connection work to promptfoo-provider-setup and adversarial work to the redteam skills. - Secrets stay in template references rather than committed values, and sharing is disabled with an explicit flag. - Install goes through the Claude Code and Codex marketplaces for the four-skill bundle; no generic package installer command is printed for the single skill. ### Not established by this review - Not executed: it was not installed or run. - No eval suite was written and no eval command was run. - No provider was connected and no grader was calibrated. The review applies to the material and revision named here. A newer upstream release can change its behavior. 03 / PUT IT TO WORK ## Add a skill to your workflow. [Upstream setup instructions ↗](https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/site/docs/integrations/agent-skill.md) - Add promptfoo/promptfoo as a marketplace source and install the promptfoo bundle in Claude Code or Codex as the docs show. - Invoke the eval skill with a namespaced command when writing or running a suite. ### Before you start - A Claude Code or Codex compatible agent for marketplace commands - A configured provider target with credentials for the suite under test ### Compatibility Claude Code · OpenAI Codex ### Write and run eval suites - Read suite configs, prompts, providers, tests and references - Write eval configs, prompts, tests and datasets in the working project - Run validate and eval commands that call the configured providers ### Cost model No charge is stated for the skill text under MIT; provider inference for eval and grader calls is billed by the configured providers. 04 / FOLLOW THE EVIDENCE ## The source trail. Our notes are separate from the original resource. Check upstream before adopting a new version. - [promptfoo-evals SKILL.md at reviewed revision ↗](https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/plugins/promptfoo/skills/promptfoo-evals/SKILL.md) Checked 2026-10-15 https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/plugins/promptfoo/skills/promptfoo-evals/SKILL.md Supports: upstreamDescription, whySelected, bestFor - [Agent skill install docs at reviewed revision ↗](https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/site/docs/integrations/agent-skill.md) Checked 2026-10-15 https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/site/docs/integrations/agent-skill.md Supports: install, compatibility, limitations - [MIT LICENSE at reviewed revision ↗](https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/LICENSE) Checked 2026-10-15 https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/LICENSE Supports: license KEEP COMPARING ## Other approaches to consider. Related by category or shared topics. These are alternatives to inspect, not a measured quality order. [### Datadog Trace RCA ↗ Root-causes production AI failures from LLM traces through phased span analysis, then recommends prompt, tool, routing or retrieval changes quoted from the trace evidence.](/skills/datadog-labs-agent-observability-trace-rca/)[### Hugging Face Local Model Evaluations ↗ Chooses a local evaluation framework and inference backend, starting with a small smoke run before scaling evaluation work.](/skills/huggingface-huggingface-community-evals/)[### Agent Platform Eval Flywheel ↗ Evaluates and iterates on Google Cloud Agent Platform models and agents, with a confirmation card before every billed evaluation run.](/skills/google-agent-platform-eval-flywheel/) [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/) ## Continue your investigation - [Choose a toolkit](/tools/) - [Inspect execution hosts](/agents/) - [Check tool access](/mcp-servers/) - [Check project dependencies](/check-repo/)