EVALUATION / Anthropic
Post-hoc skill analyzer
Skill-creator agent that reads a finished blind comparison, then opens both skills and transcripts to say why the winner won.
“Runs after the comparator has chosen A or B, reads winner and loser paths, and writes improvement notes to output_path.”
01 / THE REASONING
Why this made the selection.
- The catalogue has the skill-creator grader and not this post-hoc analyzer.
- The file is explicit that it runs only after a blind winner exists, so it is not a second copy of the comparator.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- plugins/skill-creator/skills/skill-creator/agents/analyzer.md
- plugins/skill-creator/LICENSE
- Round 2: pinned definition, output contract and permission/dependency boundaries reread on 2026-10-14
Our findings
- The agent receives winner, both skill paths, both transcript paths, the comparator JSON, and an output path.
- It reads the comparator result before it reads the skills, then looks at instruction specificity, tool use, examples, and edge cases.
- The file says it unblids the comparison by reading the skills and transcripts after the blind winner is known.
- The comparison mode examines instruction, tooling, example and reference differences after seeing the winner.
- The benchmark mode forbids improvement recommendations and instead highlights nondiscriminating assertions, variability and quality/cost patterns.
Not established by this review
- The agent was not executed.
- No transcript was read with it.
- No output file was written.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Use the role in your project.
- No concrete install command without a placeholder is in the marketplace README.
- This agent is invoked by the skill-creator evaluation flow, which was not run.
- Keep the task mode explicit and retain all run evidence, failed cases and scoring definitions before interpreting an apparent winner.
Before you start
- A finished blind comparison with paths to both skills and transcripts
- Claude Code skill-creator
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
- analyzer.md ↗Checked
https://github.com/anthropics/claude-plugins-official/blob/8bd00f08f0b04705e610cf132fbe2cf5ec44cfe3/plugins/skill-creator/skills/skill-creator/agents/analyzer.md
Supports: post-hoc role and inputs
- skill-creator LICENSE ↗Checked
https://github.com/anthropics/claude-plugins-official/blob/8bd00f08f0b04705e610cf132fbe2cf5ec44cfe3/plugins/skill-creator/LICENSE
Supports: Apache-2.0