--- title: "Validate an Eval Scorer: review, setup & limitations · Undominated.ai" canonical: https://undominated.ai/skills/braintrustdata-braintrust-validate-eval-scorer/ description: "Checks an LLM judge or scorer against expert-reviewed labels, using agreement with uncertainty and severity-weighted errors." --- # Validate an Eval Scorer: review, setup & limitations · Undominated.ai > Checks an LLM judge or scorer against expert-reviewed labels, using agreement with uncertainty and severity-weighted errors. [← Explore all skills](/skills/) MODEL EVALUATION / Braintrust # Validate an Eval Scorer Checks an LLM judge or scorer against expert-reviewed labels, using agreement with uncertainty and severity-weighted errors. [See setup guidance ↓](#setup)[Original source ↗](https://github.com/braintrustdata/eval-library/tree/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-validate-eval-scorer) SOURCE REVIEW Reviewed 2026-10-13 Evidence 5 linked sources Publisher Braintrust Licence [Apache-2.0 ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE) Revision 04486b39047b [Read what was—and wasn’t—checked ↓](#review) “Validate automated eval scorers and LLM judges against expert-reviewed reference data. Use to compare scorer output with human labels, calculate agreement (kappa, alpha) with uncertainty, inspect confusion by class and severity, analyze subgroup failures, test shortcut and gaming cases, propagate scorer error into headline numbers, document blind spots, and decide whether a scorer is fit for exploration, trend monitoring, or release gating. Do not use to create the initial scorer or to design the human review workflow.” [Braintrust · upstream description ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-validate-eval-scorer/SKILL.md) Our analysis follows below. 01 / THE REASONING ## Why this made the selection. - It requires naming the reference tier first, says a strong-model reference supports iteration only, and calibrates fitness bands to adjudicated human labels. - It rejects raw accuracy, requires kappa or alpha with uncertainty, leads with the most decision-relevant false acceptance, and propagates scorer error into headline numbers. ### A good fit for - Deciding whether a judge may gate a release - Re-validating a scorer after a version bump ### Weigh up before choosing - The body cites fitness bands in reference.md, a file not present in the skill directory at this revision. The header points to references/validation-recipe.md, which was not opened. - The Braintrust section assumes Braintrust experiments and datasets; the README says the method itself is platform-independent. - The README installs by git clone plus a symlink loop, which tracks the default branch rather than this revision. - Scorer outputs and reference labels must align at both item and criterion level before agreement results can be interpreted. 02 / THE REVIEW RECORD ## What we actually inspected. Source review has boundaries. A clear record is more useful than a “safe” badge. ### Material inspected - skills/braintrust-validate-eval-scorer/SKILL.md (trigger, do, avoid, check, risk, Braintrust) - README.md (skills overview, installation, licence) - Repository tree for the skill directory - LICENSE Apache-2.0 text ### Our findings - Validating on the examples used to tune the scorer is ruled out. - The verdict must name allowed uses, prohibited uses and revalidation triggers. ### Not established by this review - The installer was not run, and the skill was not executed against a project or service. No scorer was validated and no Braintrust project was contacted. - references/interaction-contract.md, platform-mechanics.md and validation-recipe.md were not opened. - Not executed: it was not installed or run. The review applies to the material and revision named here. A newer upstream release can change its behavior. 03 / PUT IT TO WORK ## Add a skill to your workflow. [Upstream setup instructions ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md) DOCUMENTED COMMAND git clone https://github.com/braintrustdata/eval-library.git Copy command ↗ Copying does not execute this command. It may retrieve a newer version than the reviewed source. - The README clones the repository and symlinks each skills/braintrust-* directory into ~/.claude/skills or ~/.codex/skills. It also says copying one skill directory works. - Check out revision 04486b39047b5ba230f795e52f2935fef651e789 after cloning to match this review. ### Before you start - git - A Braintrust project only for the Braintrust-specific steps ### Compatibility Agent Skills-compatible hosts · Claude Code · Codex CLI ### Host-mediated analysis of local labels and scorer outputs - Reads scorer outputs and reference labels the user supplies and computes agreement statistics. - The Braintrust section runs the scorer as an experiment over a versioned dataset. ### Cost model Source is Apache-2.0. Running a judge over a golden set calls a model billed by its provider. Those charges were not measured. 04 / FOLLOW THE EVIDENCE ## The source trail. Our notes are separate from the original resource. Check upstream before adopting a new version. - [Skill definition ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-validate-eval-scorer/SKILL.md) Checked 2026-10-13 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-validate-eval-scorer/SKILL.md Supports: summary, upstreamDescription, whySelected, limitations, access, review - [Install instructions and skill index ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md) Checked 2026-10-13 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md Supports: install, compatibility - [Repository licence ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE) Checked 2026-10-13 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE Supports: license - [2026-10-15 W32 source re-review at the existing pin ↗](https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-validate-eval-scorer/SKILL.md) Checked 2026-10-15 https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-validate-eval-scorer/SKILL.md Supports: upstreamDescription, review.notTested, limitations - [2026-10-15 W32 source re-review at the existing pin ↗](https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/README.md) Checked 2026-10-15 https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/README.md Supports: install.command KEEP COMPARING ## Other approaches to consider. Related by category or shared topics. These are alternatives to inspect, not a measured quality order. [### Agent Platform Eval Flywheel ↗ Evaluates and iterates on Google Cloud Agent Platform models and agents, with a confirmation card before every billed evaluation run.](/skills/google-agent-platform-eval-flywheel/)[### Arize Evaluator ↗ Creates and runs Arize LLM-as-judge and code evaluators through the ax CLI, with a rule against fabricating scores.](/skills/arize-ai-arize-evaluator/)[### LangSmith Evaluator ↗ Builds LangSmith offline and online evaluators, with rules for name-based deletes and judge model configuration.](/skills/langchain-ai-langsmith-evaluator/) [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/) ## Continue your investigation - [Choose a toolkit](/tools/) - [Inspect execution hosts](/agents/) - [Check tool access](/mcp-servers/) - [Check project dependencies](/check-repo/)