---
title: "Braintrust Build Eval Dataset: review, setup & limitations · Undominated.ai"
canonical: https://undominated.ai/skills/braintrustdata-braintrust-build-eval-dataset/
description: "Draft or audit an evaluation dataset's population, label provenance, transform invariants and split lifecycle. This is a review procedure, not an executable dataset validator."
---

# Braintrust Build Eval Dataset: review, setup & limitations · Undominated.ai

> Draft or audit an evaluation dataset's population, label provenance, transform invariants and split lifecycle. This is a review procedure, not an executable dataset validator.

[← Explore all skills](/skills/)

AI EVALUATION / Braintrust

# Braintrust Build Eval Dataset

Draft or audit an evaluation dataset's population, label provenance, transform invariants and split lifecycle. This is a review procedure, not an executable dataset validator.

 [See setup guidance ↓](#setup)[Original source ↗](https://github.com/braintrustdata/eval-library/tree/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset)

SOURCE REVIEW

 Reviewed 2026-10-15
 Evidence 8 linked sources
 Publisher Braintrust
 Licence [Apache-2.0 ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE)
 Revision 04486b39047b
 [Read what was—and wasn’t—checked ↓](#review)

“Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev/test splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.”

 [Braintrust · upstream description ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/SKILL.md) Our analysis follows below.

01 / THE REASONING

## Why this made the selection.

 - Adds dataset construction and audit before the catalogue's existing sample-size and scorer-validation stages; it requires population and label provenance rather than treating more rows as better evidence.
- Its transform-invariant and stable-reference requirements are useful before batch inference: reject corrupted inputs and preserve which source unit produced each case.

### A good fit for

 - A dataset datasheet with source, strata, label-audit and split decisions
- A pre-inference review of truncation, redaction, deduplication and leakage risks

### Weigh up before choosing

 - Instructions do not mechanically validate data, enforce test-touch counts or prove a corpus uncontaminated. Sample-size calculations and scorer implementations are explicitly outside this skill.
- The Braintrust section can lead to dataset writes and processing of production traces. Use only authorized, minimized input; a local planning request does not authorize upload.
- The shared platform reference advises deleting smoke and partially errored experiments. Preserve the failure evidence and require separate deletion approval; this review did not authorize that action.
- The card says reference.md in several places, but the directory contains references/dataset-construction.md instead. Its trace-row-shape reference also points to a different skill's directory; a single-directory copy does not include that extra reference.

02 / THE REVIEW RECORD

## What we actually inspected.

Source review has boundaries. A clear record is more useful than a “safe” badge.

### Material inspected

 - Pinned SKILL.md and all three files in its references directory
- Cross-skill references/facet-pipeline.md for trace, span and group row shapes
- Root README, skills/README.md, LICENSE and recursive Git tree

### Our findings

 - The definition separates production, expert-authored, synthetic and public-benchmark sources, requires per-item label provenance, and treats synthetic data as a documented gap filler rather than population evidence.
- The dataset reference calls for an invariant after every transform, rejects corrupted rows, versions the transformation pipeline, and keeps dev/test separation and test-touch counts explicit.
- The Braintrust section stores trace_ref, group_ref or _full_row_id_ref instead of derived trace text; the linked facet reference supplies those row shapes. Immutable source availability still needs checking separately.
- The interaction contract explicitly treats traces and model outputs as data, not instructions, and says audit must not silently become repair.
- The selected directory contains only Markdown: no install hook, executable helper, MCP handler or permission-enforcing config. The publisher's manual directory-copy route avoids its separate runnable examples.
- The inspected Apache-2.0 text grants redistribution subject to licence and notice obligations. No source bundle or licence bytes are redistributed by this record.

### Not established by this review

 - No skill installation or agent activation
- No dataset upload, row transformation, label audit, scorer run or Braintrust API call
- No verification of the cited statistical literature or current Braintrust product mechanics
- Not executed: it was not installed or run.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

## Add a skill to your workflow.

[Upstream setup instructions ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md)
 - The publisher documents copying an individual skill directory, including references, into a compatible agent's skill location; this review applies to the displayed commit.
- Keep the root Apache licence with any authorized copy. Read the separately linked facet-pipeline reference if using its trace row shapes.
- Enable only the dataset review needed for the task. Approve data access, writes, uploads and deletions independently; no installation was performed here.

### Before you start

 - An Agent Skills-compatible host; publisher names Claude Code and Codex CLI
- Authorized example cases and a defined intended population
- Braintrust project access only if choosing the optional Braintrust-specific operations; no account needed to read the methodology

### Compatibility

Agent Skills · Claude Code · Codex CLI

### Implementation

Markdown

### Local instructions; optional authenticated dataset operations

 - Read only the supplied cases, schema and authorized trace exports for audit
- Write a datasheet or revised dataset only when requested
- Separate authorization for Braintrust project reads, dataset writes and experiment deletion; no credential scope is enforced by the Markdown

### Cost model

No fee to read the Apache-licensed definition; host/model usage and optional hosted operations may be billable. No current rate verified.

04 / FOLLOW THE EVIDENCE

## The source trail.

Our notes are separate from the original resource. Check upstream before adopting a new version.

 - [Pinned dataset definition ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/SKILL.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/SKILL.md Supports: Description excerpt, Scope, data lifecycle and stable source references
- [Dataset construction reference ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/dataset-construction.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/dataset-construction.md Supports: Label audit, transform invariants, leakage caveats and dev/test boundaries
- [Interaction contract ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/interaction-contract.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/interaction-contract.md Supports: Audit versus repair and untrusted trace-content boundary
- [Platform mechanics ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/platform-mechanics.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/references/platform-mechanics.md Supports: Version pinning, pagination, local caching and deletion instruction
- [Cross-skill row-shape reference ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-discover-trace-topics/references/facet-pipeline.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-discover-trace-topics/references/facet-pipeline.md Supports: Referenced span, trace and group shapes; not a bundled executable
- [Publisher installation instructions ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md Supports: Manual individual-directory copy and named hosts
- [Pinned licence ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE Supports: Apache-2.0 redistribution conditions
- [2026-10-15 W28 source re-review at the existing pin ↗](https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/SKILL.md) Checked 2026-10-15 https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-build-eval-dataset/SKILL.md Supports: upstreamDescription, review.notTested, review.reviewedAt

KEEP COMPARING

## Other approaches to consider.

Related by category or shared topics. These are alternatives to inspect, not a measured quality order.

 [### Braintrust Define Eval Release Gate ↗ Translate completed, intact evaluation evidence into owned ship-or-hold rules. Separates improvement targets from non-negotiable guardrails; does not implement or activate CI.](/skills/braintrustdata-braintrust-define-eval-release-gate/)[### Braintrust Design Eval Experiment ↗ Prepare a comparison manifest before inference: freeze the serving path, tool permissions, pairing, treatment implementation and stopping rules. It does not run an experiment.](/skills/braintrustdata-braintrust-design-eval-experiment/)[### Undominated · System prompt extraction guard audit ↗ Audits resistance against system prompt leakage attacks and multi-turn extraction attempts.](/skills/undominated-system-prompt-extraction-guard-audit/)

 [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/)

## Continue your investigation

 - [Choose a toolkit](/tools/)
- [Inspect execution hosts](/agents/)
- [Check tool access](/mcp-servers/)
- [Check project dependencies](/check-repo/)
