---
title: "Size an Eval Dataset: review, setup & limitations · Undominated.ai"
canonical: https://undominated.ai/skills/braintrustdata-braintrust-size-eval-dataset/
description: "Computes or audits eval sample sizes, detectable effects and clean-trial counts, with clustering and pairing assumptions stated."
---

# Size an Eval Dataset: review, setup & limitations · Undominated.ai

> Computes or audits eval sample sizes, detectable effects and clean-trial counts, with clustering and pairing assumptions stated.

[← Explore all skills](/skills/)

MODEL EVALUATION / Braintrust

# Size an Eval Dataset

Computes or audits eval sample sizes, detectable effects and clean-trial counts, with clustering and pairing assumptions stated.

 [See setup guidance ↓](#setup)[Original source ↗](https://github.com/braintrustdata/eval-library/tree/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-size-eval-dataset)

SOURCE REVIEW

 Reviewed 2026-10-13
 Evidence 5 linked sources
 Publisher Braintrust
 Licence [Apache-2.0 ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE)
 Revision 04486b39047b
 [Read what was—and wasn’t—checked ↓](#review)

“Calculate or audit eval sample sizes, minimum detectable effects, confidence interval precision, required repeated runs, and clean-trial counts for bounding rare failures. Use when a user asks how many eval cases, items, scenarios, runs, or safety trials are needed, whether an existing dataset is adequately powered, whether N examples can detect an X-point gain, or how many clean trials certify a low violation rate. Account for paired designs, clustering, target confidence, and practical effect size. Do not use for general dataset composition.”

 [Braintrust · upstream description ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-size-eval-dataset/SKILL.md) Our analysis follows below.

01 / THE REASONING

## Why this made the selection.

 - It forbids returning a sample size without stating effect size, baseline rate, risk tolerance, clustering, pairing, confidence and power, and requires a sensitivity range rather than one number.
- It rules out the normal approximation near 0 or 1 in favour of Wilson, and says a smoke run over a handful of items verifies plumbing only.

### A good fit for

 - Checking whether N examples can detect a stated gain
- Sizing clean trials to bound a rare failure rate
- Budget an evaluation across items, comparison arms and repeated runs before checking quotas and credit balances.

### Weigh up before choosing

 - The body cites anchors in reference.md, which is not in the skill directory at this revision. The header points to references/power-analysis.md, which was not opened.
- The minimum meaningful effect is taken from a release gate, so the skill assumes one exists or asks for it.
- The README installs by git clone plus a symlink loop, which tracks the default branch rather than this revision.

02 / THE REVIEW RECORD

## What we actually inspected.

Source review has boundaries. A clear record is more useful than a “safe” badge.

### Material inspected

 - skills/braintrust-size-eval-dataset/SKILL.md (trigger, do, avoid, check, risk, Braintrust)
- README.md (skills overview, installation, licence)
- Repository tree for the skill directory
- LICENSE Apache-2.0 text

### Our findings

 - Gating runs use K >= 3 recorded trials, and the worst run is what a latency or reliability gate reads, per the file.
- A design effect is applied for clustered or repeated-variant items, which the file calls the largest single source of overstated precision.

### Not established by this review

 - The installer was not run, and the skill was not executed against a project or service. No sample size was computed.
- references/interaction-contract.md, platform-mechanics.md and power-analysis.md were not opened.
- Not executed: it was not installed or run.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

## Add a skill to your workflow.

[Upstream setup instructions ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md)

DOCUMENTED COMMAND

 git clone https://github.com/braintrustdata/eval-library.git Copy command ↗

Copying does not execute this command. It may retrieve a newer version than the reviewed source.

 - The README clones the repository and symlinks each skills/braintrust-* directory into ~/.claude/skills or ~/.codex/skills. It also says copying one skill directory works.
- Check out revision 04486b39047b5ba230f795e52f2935fef651e789 after cloning to match this review.

### Before you start

 - git
- A Braintrust project only for the Braintrust-specific steps

### Compatibility

Agent Skills-compatible hosts · Claude Code · Codex CLI

### Host-mediated calculation from user-supplied assumptions

 - Computes sample sizes from the effect, rate and design assumptions the user supplies.
- Checks the items x arms x runs product against quotas and credit balances before a run, per the file.

### Cost model

Source is Apache-2.0. The calculation itself needs no paid service; the eval runs it sizes are billed by their providers. Those charges were not measured.

04 / FOLLOW THE EVIDENCE

## The source trail.

Our notes are separate from the original resource. Check upstream before adopting a new version.

 - [Skill definition ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-size-eval-dataset/SKILL.md) Checked 2026-10-13 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-size-eval-dataset/SKILL.md Supports: summary, upstreamDescription, whySelected, limitations, access, review
- [Install instructions and skill index ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md) Checked 2026-10-13 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md Supports: install, compatibility
- [Repository licence ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE) Checked 2026-10-13 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE Supports: license
- [2026-10-15 W31 source re-review at the existing pin ↗](https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-size-eval-dataset/SKILL.md) Checked 2026-10-15 https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-size-eval-dataset/SKILL.md Supports: upstreamDescription, review.notTested, bestFor
- [2026-10-15 W31 source re-review at the existing pin ↗](https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/README.md) Checked 2026-10-15 https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/README.md Supports: install.command

KEEP COMPARING

## Other approaches to consider.

Related by category or shared topics. These are alternatives to inspect, not a measured quality order.

 [### Agent Platform Eval Flywheel ↗ Evaluates and iterates on Google Cloud Agent Platform models and agents, with a confirmation card before every billed evaluation run.](/skills/google-agent-platform-eval-flywheel/)[### Arize Evaluator ↗ Creates and runs Arize LLM-as-judge and code evaluators through the ax CLI, with a rule against fabricating scores.](/skills/arize-ai-arize-evaluator/)[### LangSmith Evaluator ↗ Builds LangSmith offline and online evaluators, with rules for name-based deletes and judge model configuration.](/skills/langchain-ai-langsmith-evaluator/)

 [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/)

## Continue your investigation

 - [Choose a toolkit](/tools/)
- [Inspect execution hosts](/agents/)
- [Check tool access](/mcp-servers/)
- [Check project dependencies](/check-repo/)
