EVALUATION / Anthropic
Skill eval grader
Grades each expectation pass or fail from cited evidence, treats uncertainty as a failure, and writes grading.json.
“Evaluate expectations against an execution transcript and outputs.”
01 / THE REASONING
Why this made the selection.
- The inputs are expectations, a transcript path, and an outputs directory, and each expectation is pass or fail.
- The grader is also told to flag an assertion that would pass for a clearly wrong output.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- plugins/skill-creator/skills/skill-creator/agents/grader.md
- plugins/skill-creator/LICENSE
- Definition reread at the preserved revision; scoped workflow and output-contract review on 2026-10-14
Our findings
- The file is titled Grader Agent and has no YAML frontmatter.
- Required prompt parameters are expectations, transcript_path, and outputs_dir.
- Each expectation is pass or fail, with no partial credit.
- The output includes evidence for the verdict, and eval_feedback only when the assertions have a clear gap.
- The grader checks actual relevant output files rather than relying only on executor descriptions, and treats unverifiable expectations as failures.
- It distinguishes copied executor metrics/timing from its own judgments and asks for assertion gaps that could let wrong outputs pass.
Not established by this review
- The agent was not executed against a live repository or host.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Use the role in your project.
- Copy grader.md at the reviewed commit into the location your host uses for agent prompts.
- No install command is published. Installing the whole skill-creator plugin was not reviewed.
Before you start
- An execution transcript
- An outputs directory
- A list of expectations
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
- Grader prompt ↗Checked
https://github.com/anthropics/claude-plugins-official/blob/b78ac49cdc6b3d7b61c4439470e311f4291265b1/plugins/skill-creator/skills/skill-creator/agents/grader.md
Supports: summary, whySelected, output format
- Apache-2.0 licence ↗Checked
https://github.com/anthropics/claude-plugins-official/blob/b78ac49cdc6b3d7b61c4439470e311f4291265b1/plugins/skill-creator/LICENSE
Supports: Redistribution terms, Unfilled copyright appendix