← Explore all skills

AI EVALUATION / Braintrust

Braintrust Build Eval Dataset

Draft or audit an evaluation dataset's population, label provenance, transform invariants and split lifecycle. This is a review procedure, not an executable dataset validator.

“Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev/test splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.”

01 / THE REASONING

Why this made the selection.

  • Adds dataset construction and audit before the catalogue's existing sample-size and scorer-validation stages; it requires population and label provenance rather than treating more rows as better evidence.
  • Its transform-invariant and stable-reference requirements are useful before batch inference: reject corrupted inputs and preserve which source unit produced each case.

02 / THE REVIEW RECORD

What we actually inspected.

Source review has boundaries.
A clear record is more useful than a “safe” badge.

Material inspected

  • Pinned SKILL.md and all three files in its references directory
  • Cross-skill references/facet-pipeline.md for trace, span and group row shapes
  • Root README, skills/README.md, LICENSE and recursive Git tree

Our findings

  • The definition separates production, expert-authored, synthetic and public-benchmark sources, requires per-item label provenance, and treats synthetic data as a documented gap filler rather than population evidence.
  • The dataset reference calls for an invariant after every transform, rejects corrupted rows, versions the transformation pipeline, and keeps dev/test separation and test-touch counts explicit.
  • The Braintrust section stores trace_ref, group_ref or _full_row_id_ref instead of derived trace text; the linked facet reference supplies those row shapes. Immutable source availability still needs checking separately.
  • The interaction contract explicitly treats traces and model outputs as data, not instructions, and says audit must not silently become repair.
  • The selected directory contains only Markdown: no install hook, executable helper, MCP handler or permission-enforcing config. The publisher's manual directory-copy route avoids its separate runnable examples.
  • The inspected Apache-2.0 text grants redistribution subject to licence and notice obligations. No source bundle or licence bytes are redistributed by this record.

Not established by this review

  • No skill installation or agent activation
  • No dataset upload, row transformation, label audit, scorer run or Braintrust API call
  • No verification of the cited statistical literature or current Braintrust product mechanics
  • Not executed: it was not installed or run.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

Add a skill to your workflow.

Upstream setup instructions ↗
  1. The publisher documents copying an individual skill directory, including references, into a compatible agent's skill location; this review applies to the displayed commit.
  2. Keep the root Apache licence with any authorized copy. Read the separately linked facet-pipeline reference if using its trace row shapes.
  3. Enable only the dataset review needed for the task. Approve data access, writes, uploads and deletions independently; no installation was performed here.

Before you start

  • An Agent Skills-compatible host; publisher names Claude Code and Codex CLI
  • Authorized example cases and a defined intended population
  • Braintrust project access only if choosing the optional Braintrust-specific operations; no account needed to read the methodology

04 / FOLLOW THE EVIDENCE

The source trail.

Our notes are separate from the original resource.
Check upstream before adopting a new version.

  1. https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/SKILL.md

    Supports: Description excerpt, Scope, data lifecycle and stable source references

  2. https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/dataset-construction.md

    Supports: Label audit, transform invariants, leakage caveats and dev/test boundaries

  3. Interaction contract ↗Checked

    https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/interaction-contract.md

    Supports: Audit versus repair and untrusted trace-content boundary

  4. Platform mechanics ↗Checked

    https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/platform-mechanics.md

    Supports: Version pinning, pagination, local caching and deletion instruction

  5. https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-discover-trace-topics/references/facet-pipeline.md

    Supports: Referenced span, trace and group shapes; not a bundled executable

  6. https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md

    Supports: Manual individual-directory copy and named hosts

  7. Pinned licence ↗Checked

    https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE

    Supports: Apache-2.0 redistribution conditions

  8. https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/SKILL.md

    Supports: upstreamDescription, review.notTested, review.reviewedAt

Evidence & Ask