AI EVALUATION / Braintrust
Braintrust Build Eval Dataset
Draft or audit an evaluation dataset's population, label provenance, transform invariants and split lifecycle. This is a review procedure, not an executable dataset validator.
“Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev/test splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.”
01 / THE REASONING
Why this made the selection.
- Adds dataset construction and audit before the catalogue's existing sample-size and scorer-validation stages; it requires population and label provenance rather than treating more rows as better evidence.
- Its transform-invariant and stable-reference requirements are useful before batch inference: reject corrupted inputs and preserve which source unit produced each case.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- Pinned SKILL.md and all three files in its references directory
- Cross-skill references/facet-pipeline.md for trace, span and group row shapes
- Root README, skills/README.md, LICENSE and recursive Git tree
Our findings
- The definition separates production, expert-authored, synthetic and public-benchmark sources, requires per-item label provenance, and treats synthetic data as a documented gap filler rather than population evidence.
- The dataset reference calls for an invariant after every transform, rejects corrupted rows, versions the transformation pipeline, and keeps dev/test separation and test-touch counts explicit.
- The Braintrust section stores trace_ref, group_ref or _full_row_id_ref instead of derived trace text; the linked facet reference supplies those row shapes. Immutable source availability still needs checking separately.
- The interaction contract explicitly treats traces and model outputs as data, not instructions, and says audit must not silently become repair.
- The selected directory contains only Markdown: no install hook, executable helper, MCP handler or permission-enforcing config. The publisher's manual directory-copy route avoids its separate runnable examples.
- The inspected Apache-2.0 text grants redistribution subject to licence and notice obligations. No source bundle or licence bytes are redistributed by this record.
Not established by this review
- No skill installation or agent activation
- No dataset upload, row transformation, label audit, scorer run or Braintrust API call
- No verification of the cited statistical literature or current Braintrust product mechanics
- Not executed: it was not installed or run.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Add a skill to your workflow.
- The publisher documents copying an individual skill directory, including references, into a compatible agent's skill location; this review applies to the displayed commit.
- Keep the root Apache licence with any authorized copy. Read the separately linked facet-pipeline reference if using its trace row shapes.
- Enable only the dataset review needed for the task. Approve data access, writes, uploads and deletions independently; no installation was performed here.
Before you start
- An Agent Skills-compatible host; publisher names Claude Code and Codex CLI
- Authorized example cases and a defined intended population
- Braintrust project access only if choosing the optional Braintrust-specific operations; no account needed to read the methodology
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
- Pinned dataset definition ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/SKILL.md
Supports: Description excerpt, Scope, data lifecycle and stable source references
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/dataset-construction.md
Supports: Label audit, transform invariants, leakage caveats and dev/test boundaries
- Interaction contract ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/interaction-contract.md
Supports: Audit versus repair and untrusted trace-content boundary
- Platform mechanics ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/platform-mechanics.md
Supports: Version pinning, pagination, local caching and deletion instruction
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-discover-trace-topics/references/facet-pipeline.md
Supports: Referenced span, trace and group shapes; not a bundled executable
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md
Supports: Manual individual-directory copy and named hosts
- Pinned licence ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE
Supports: Apache-2.0 redistribution conditions
https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/SKILL.md
Supports: upstreamDescription, review.notTested, review.reviewedAt