MODEL EVALUATION / Promptfoo
Promptfoo evals
Official Promptfoo skill that writes, runs and refines declarative eval suites with deterministic assertions first and a CI gate.
“Write, run, and improve non-redteam Promptfoo eval suites for a configured target: test cases, assertions, rubrics, datasets, and CI gates. Use promptfoo-provider-setup first for a new or broken connection; use the redteam skills for adversarial scans.”
01 / THE REASONING
Why this made the selection.
- Defines a non-redteam eval workflow with deterministic assertions before model-graded rubrics, inlined source evidence, calibrated known-good and known-bad controls, and an exit-code CI gate.
- Declares explicit handoffs to the sibling provider-setup skill for new or broken connections and to the redteam skills for adversarial scans, instead of covering everything in one file.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- plugins/promptfoo/skills/promptfoo-evals/SKILL.md
- plugins/promptfoo/skills/promptfoo-evals/references/eval-patterns.md
- site/docs/integrations/agent-skill.md
- LICENSE
Our findings
- The skill routes provider connection work to promptfoo-provider-setup and adversarial work to the redteam skills.
- Secrets stay in template references rather than committed values, and sharing is disabled with an explicit flag.
- Install goes through the Claude Code and Codex marketplaces for the four-skill bundle; no generic package installer command is printed for the single skill.
Not established by this review
- Not executed: it was not installed or run.
- No eval suite was written and no eval command was run.
- No provider was connected and no grader was calibrated.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Add a skill to your workflow.
- Add promptfoo/promptfoo as a marketplace source and install the promptfoo bundle in Claude Code or Codex as the docs show.
- Invoke the eval skill with a namespaced command when writing or running a suite.
Before you start
- A Claude Code or Codex compatible agent for marketplace commands
- A configured provider target with credentials for the suite under test
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/plugins/promptfoo/skills/promptfoo-evals/SKILL.md
Supports: upstreamDescription, whySelected, bestFor
https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/site/docs/integrations/agent-skill.md
Supports: install, compatibility, limitations
https://github.com/promptfoo/promptfoo/blob/8570ed47327c7866beba017144b6197a720c96f4/LICENSE
Supports: license