MODEL EVALUATION / Braintrust
Size an Eval Dataset
Computes or audits eval sample sizes, detectable effects and clean-trial counts, with clustering and pairing assumptions stated.
“Calculate or audit eval sample sizes, minimum detectable effects, confidence interval precision, required repeated runs, and clean-trial counts for bounding rare failures. Use when a user asks how many eval cases, items, scenarios, runs, or safety trials are needed, whether an existing dataset is adequately powered, whether N examples can detect an X-point gain, or how many clean trials certify a low violation rate. Account for paired designs, clustering, target confidence, and practical effect size. Do not use for general dataset composition.”
01 / THE REASONING
Why this made the selection.
- It forbids returning a sample size without stating effect size, baseline rate, risk tolerance, clustering, pairing, confidence and power, and requires a sensitivity range rather than one number.
- It rules out the normal approximation near 0 or 1 in favour of Wilson, and says a smoke run over a handful of items verifies plumbing only.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- skills/braintrust-size-eval-dataset/SKILL.md (trigger, do, avoid, check, risk, Braintrust)
- README.md (skills overview, installation, licence)
- Repository tree for the skill directory
- LICENSE Apache-2.0 text
Our findings
- Gating runs use K >= 3 recorded trials, and the worst run is what a latency or reliability gate reads, per the file.
- A design effect is applied for clustered or repeated-variant items, which the file calls the largest single source of overstated precision.
Not established by this review
- The installer was not run, and the skill was not executed against a project or service. No sample size was computed.
- references/interaction-contract.md, platform-mechanics.md and power-analysis.md were not opened.
- Not executed: it was not installed or run.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Add a skill to your workflow.
DOCUMENTED COMMAND
git clone https://github.com/braintrustdata/eval-library.git Copying does not execute this command. It may retrieve a newer version than the reviewed source.
- The README clones the repository and symlinks each skills/braintrust-* directory into ~/.claude/skills or ~/.codex/skills. It also says copying one skill directory works.
- Check out revision 04486b39047b5ba230f795e52f2935fef651e789 after cloning to match this review.
Before you start
- git
- A Braintrust project only for the Braintrust-specific steps
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
- Skill definition ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-size-eval-dataset/SKILL.md
Supports: summary, upstreamDescription, whySelected, limitations, access, review
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md
Supports: install, compatibility
- Repository licence ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE
Supports: license
https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-size-eval-dataset/SKILL.md
Supports: upstreamDescription, review.notTested, bestFor
https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/README.md
Supports: install.command