← Explore all skills

MODEL EVALUATION / Braintrust

Size an Eval Dataset

Computes or audits eval sample sizes, detectable effects and clean-trial counts, with clustering and pairing assumptions stated.

“Calculate or audit eval sample sizes, minimum detectable effects, confidence interval precision, required repeated runs, and clean-trial counts for bounding rare failures. Use when a user asks how many eval cases, items, scenarios, runs, or safety trials are needed, whether an existing dataset is adequately powered, whether N examples can detect an X-point gain, or how many clean trials certify a low violation rate. Account for paired designs, clustering, target confidence, and practical effect size. Do not use for general dataset composition.”

01 / THE REASONING

Why this made the selection.

  • It forbids returning a sample size without stating effect size, baseline rate, risk tolerance, clustering, pairing, confidence and power, and requires a sensitivity range rather than one number.
  • It rules out the normal approximation near 0 or 1 in favour of Wilson, and says a smoke run over a handful of items verifies plumbing only.

02 / THE REVIEW RECORD

What we actually inspected.

Source review has boundaries.
A clear record is more useful than a “safe” badge.

Material inspected

  • skills/braintrust-size-eval-dataset/SKILL.md (trigger, do, avoid, check, risk, Braintrust)
  • README.md (skills overview, installation, licence)
  • Repository tree for the skill directory
  • LICENSE Apache-2.0 text

Our findings

  • Gating runs use K >= 3 recorded trials, and the worst run is what a latency or reliability gate reads, per the file.
  • A design effect is applied for clustered or repeated-variant items, which the file calls the largest single source of overstated precision.

Not established by this review

  • The installer was not run, and the skill was not executed against a project or service. No sample size was computed.
  • references/interaction-contract.md, platform-mechanics.md and power-analysis.md were not opened.
  • Not executed: it was not installed or run.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

Add a skill to your workflow.

Upstream setup instructions ↗

DOCUMENTED COMMAND

git clone https://github.com/braintrustdata/eval-library.git

Copying does not execute this command. It may retrieve a newer version than the reviewed source.

  1. The README clones the repository and symlinks each skills/braintrust-* directory into ~/.claude/skills or ~/.codex/skills. It also says copying one skill directory works.
  2. Check out revision 04486b39047b5ba230f795e52f2935fef651e789 after cloning to match this review.

Before you start

  • git
  • A Braintrust project only for the Braintrust-specific steps

04 / FOLLOW THE EVIDENCE

The source trail.

Our notes are separate from the original resource.
Check upstream before adopting a new version.

  1. Skill definition ↗Checked

    https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-size-eval-dataset/SKILL.md

    Supports: summary, upstreamDescription, whySelected, limitations, access, review

  2. https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md

    Supports: install, compatibility

  3. Repository licence ↗Checked

    https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE

    Supports: license

  4. https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-size-eval-dataset/SKILL.md

    Supports: upstreamDescription, review.notTested, bestFor

  5. https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/README.md

    Supports: install.command

Evidence & Ask