← Explore all agents

EVALUATION / Anthropic

Skill eval grader

Grades each expectation pass or fail from cited evidence, treats uncertainty as a failure, and writes grading.json.

“Evaluate expectations against an execution transcript and outputs.”

01 / THE REASONING

Why this made the selection.

  • The inputs are expectations, a transcript path, and an outputs directory, and each expectation is pass or fail.
  • The grader is also told to flag an assertion that would pass for a clearly wrong output.

02 / THE REVIEW RECORD

What we actually inspected.

Source review has boundaries.
A clear record is more useful than a “safe” badge.

Material inspected

  • plugins/skill-creator/skills/skill-creator/agents/grader.md
  • plugins/skill-creator/LICENSE
  • Definition reread at the preserved revision; scoped workflow and output-contract review on 2026-10-14

Our findings

  • The file is titled Grader Agent and has no YAML frontmatter.
  • Required prompt parameters are expectations, transcript_path, and outputs_dir.
  • Each expectation is pass or fail, with no partial credit.
  • The output includes evidence for the verdict, and eval_feedback only when the assertions have a clear gap.
  • The grader checks actual relevant output files rather than relying only on executor descriptions, and treats unverifiable expectations as failures.
  • It distinguishes copied executor metrics/timing from its own judgments and asks for assertion gaps that could let wrong outputs pass.

Not established by this review

  • The agent was not executed against a live repository or host.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

Use the role in your project.

Upstream setup instructions ↗
  1. Copy grader.md at the reviewed commit into the location your host uses for agent prompts.
  2. No install command is published. Installing the whole skill-creator plugin was not reviewed.

Before you start

  • An execution transcript
  • An outputs directory
  • A list of expectations

04 / FOLLOW THE EVIDENCE

The source trail.

Our notes are separate from the original resource.
Check upstream before adopting a new version.

  1. Grader prompt ↗Checked

    https://github.com/anthropics/claude-plugins-official/blob/b78ac49cdc6b3d7b61c4439470e311f4291265b1/plugins/skill-creator/skills/skill-creator/agents/grader.md

    Supports: summary, whySelected, output format

  2. Apache-2.0 licence ↗Checked

    https://github.com/anthropics/claude-plugins-official/blob/b78ac49cdc6b3d7b61c4439470e311f4291265b1/plugins/skill-creator/LICENSE

    Supports: Redistribution terms, Unfilled copyright appendix

Evidence & Ask