MODEL EVALUATION / Braintrust
Validate an Eval Scorer
Checks an LLM judge or scorer against expert-reviewed labels, using agreement with uncertainty and severity-weighted errors.
“Validate automated eval scorers and LLM judges against expert-reviewed reference data. Use to compare scorer output with human labels, calculate agreement (kappa, alpha) with uncertainty, inspect confusion by class and severity, analyze subgroup failures, test shortcut and gaming cases, propagate scorer error into headline numbers, document blind spots, and decide whether a scorer is fit for exploration, trend monitoring, or release gating. Do not use to create the initial scorer or to design the human review workflow.”
01 / THE REASONING
Why this made the selection.
- It requires naming the reference tier first, says a strong-model reference supports iteration only, and calibrates fitness bands to adjudicated human labels.
- It rejects raw accuracy, requires kappa or alpha with uncertainty, leads with the most decision-relevant false acceptance, and propagates scorer error into headline numbers.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- skills/braintrust-validate-eval-scorer/SKILL.md (trigger, do, avoid, check, risk, Braintrust)
- README.md (skills overview, installation, licence)
- Repository tree for the skill directory
- LICENSE Apache-2.0 text
Our findings
- Validating on the examples used to tune the scorer is ruled out.
- The verdict must name allowed uses, prohibited uses and revalidation triggers.
Not established by this review
- The installer was not run, and the skill was not executed against a project or service. No scorer was validated and no Braintrust project was contacted.
- references/interaction-contract.md, platform-mechanics.md and validation-recipe.md were not opened.
- Not executed: it was not installed or run.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Add a skill to your workflow.
DOCUMENTED COMMAND
git clone https://github.com/braintrustdata/eval-library.gitCopying does not execute this command. It may retrieve a newer version than the reviewed source.
- The README clones the repository and symlinks each skills/braintrust-* directory into ~/.claude/skills or ~/.codex/skills. It also says copying one skill directory works.
- Check out revision 04486b39047b5ba230f795e52f2935fef651e789 after cloning to match this review.
Before you start
- git
- A Braintrust project only for the Braintrust-specific steps
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
- Skill definition ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-validate-eval-scorer/SKILL.md
Supports: summary, upstreamDescription, whySelected, limitations, access, review
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md
Supports: install, compatibility
- Repository licence ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE
Supports: license
https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-validate-eval-scorer/SKILL.md
Supports: upstreamDescription, review.notTested, limitations
https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/README.md
Supports: install.command