---
title: "Braintrust Design Eval Experiment: review, setup & limitations · Undominated.ai"
canonical: https://undominated.ai/skills/braintrustdata-braintrust-design-eval-experiment/
description: "Prepare a comparison manifest before inference: freeze the serving path, tool permissions, pairing, treatment implementation and stopping rules. It does not run an experiment."
---

# Braintrust Design Eval Experiment: review, setup & limitations · Undominated.ai

> Prepare a comparison manifest before inference: freeze the serving path, tool permissions, pairing, treatment implementation and stopping rules. It does not run an experiment.

[← Explore all skills](/skills/)

AI EVALUATION / Braintrust

# Braintrust Design Eval Experiment

Prepare a comparison manifest before inference: freeze the serving path, tool permissions, pairing, treatment implementation and stopping rules. It does not run an experiment.

 [See setup guidance ↓](#setup)[Original source ↗](https://github.com/braintrustdata/eval-library/tree/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment)

SOURCE REVIEW

 Reviewed 2026-10-15
 Evidence 7 linked sources
 Publisher Braintrust
 Licence [Apache-2.0 ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE)
 Revision 04486b39047b
 [Read what was—and wasn’t—checked ↓](#review)

“Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected.”

 [Braintrust · upstream description ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/SKILL.md) Our analysis follows below.

01 / THE REASONING

## Why this made the selection.

 - Makes batching, provider, region, cache state and decoding part of the controlled comparison instead of treating a model name as a reproducible configuration.
- Requires per-arm tool manifests and implementable treatments, complementing existing scorer and dataset-size reviews without duplicating them.

### A good fit for

 - A pre-analysis manifest for a batch-inference or model-switch comparison
- A reproducibility review before spending inference quota

### Weigh up before choosing

 - No batch submission, retry implementation, quota enforcement or provider integration is supplied. The manifest is a plan, not a run receipt.
- Illustrative hypotheses and repetition defaults require justification; they are not measured guarantees or ready-made acceptance thresholds.
- Platform-specific pinning, naming and pagination statements were inspected as publisher guidance, not tested against a live Braintrust account.
- The shared reference instructs deletion of smoke and partially errored experiments. Keep evidence and require separate approval; never silently remove failures to improve a comparison.
- A reference.md pointer in the card has no matching file; the actual design reference is references/experiment-manifest.md.

02 / THE REVIEW RECORD

## What we actually inspected.

Source review has boundaries. A clear record is more useful than a “safe” badge.

### Material inspected

 - Pinned definition and references/experiment-manifest.md
- Its interaction-contract.md and platform-mechanics.md
- Root README, LICENSE and recursive Git tree

### Our findings

 - The manifest reference records provider, endpoint tier, region, decoding, cache state and batching, and asks for tool names, schemas and permissions per arm.
- It excludes an arm that cannot receive a treatment from the treatment-effect claim rather than labeling that arm as receiving no benefit.
- It fixes stopping, exclusion and multiplicity rules before collection, distinguishes exploratory from confirmatory runs, and says to restart or disclose a defect instead of changing configuration mid-comparison.
- Run-integrity guidance requires a concurrency cap, a whole-matrix quota budget and separately recorded trials. These are requirements for a future runner, not mechanisms in this Markdown.
- The selected directory contains a definition and three Markdown references, without executable helpers, lifecycle hooks, MCP handlers or an enforcement config. The root manual-copy route is independent of runnable example projects.
- The common contract distinguishes audit from repair and treats trace content as evidence. Apache-2.0 licence terms were read at the same commit.

### Not established by this review

 - No skill installation or host discovery
- No inference request, paired run, quota probe, experiment creation or deletion
- No statistical computation or live verification of Braintrust platform claims
- Not executed: it was not installed or run.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

## Add a skill to your workflow.

[Upstream setup instructions ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md)
 - Use the publisher's individual-directory copy option with the pinned design skill and its references, preserving licence notices.
- Load it in a compatible host for manifest authoring; do not install the repository's separate runnable study examples as a prerequisite.
- Keep all account changes and future execution behind task-specific approval. No install command or host configuration was composed for this record.

### Before you start

 - Agent Skills-compatible host; Claude Code and Codex CLI are named by the publisher
- Proposed arms, intended population and available dataset/scorer identifiers
- No cloud credentials needed for local manifest authoring; authenticated project access only for optional platform changes

### Compatibility

Agent Skills · Claude Code · Codex CLI

### Implementation

Markdown

### Planning instructions; optional experiment metadata writes

 - Read approved configuration, dataset and scorer metadata
- Write a local experiment manifest when requested
- Provider calls, Braintrust experiment writes or deletion need separate authorization and credentials; the skill supplies no credential isolation

### Cost model

Definition is Apache-licensed; agent usage and any later inference or hosted experiment operations may incur costs. No rates measured.

04 / FOLLOW THE EVIDENCE

## The source trail.

Our notes are separate from the original resource. Check upstream before adopting a new version.

 - [Pinned experiment-design definition ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/SKILL.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/SKILL.md Supports: Exact description excerpt and pre-run scope
- [Experiment manifest reference ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/references/experiment-manifest.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/references/experiment-manifest.md Supports: Batching and serving controls, tool permissions, quotas and treatment implementation
- [Interaction contract ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/references/interaction-contract.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/references/interaction-contract.md Supports: Uncertainty labels, audit boundaries and trace trust boundary
- [Platform mechanics ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/references/platform-mechanics.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/references/platform-mechanics.md Supports: Pinning, naming, caching and deletion limitations
- [Publisher installation instructions ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md Supports: Manual skill-directory copy and supported host claims
- [Pinned licence ↗](https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE) Checked 2026-10-14 https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE Supports: Apache-2.0 licence inspection
- [2026-10-15 W30 source re-review at the existing pin ↗](https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/SKILL.md) Checked 2026-10-15 https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/SKILL.md Supports: upstreamDescription, review.notTested, review.reviewedAt

KEEP COMPARING

## Other approaches to consider.

Related by category or shared topics. These are alternatives to inspect, not a measured quality order.

 [### Braintrust Build Eval Dataset ↗ Draft or audit an evaluation dataset's population, label provenance, transform invariants and split lifecycle. This is a review procedure, not an executable dataset validator.](/skills/braintrustdata-braintrust-build-eval-dataset/)[### Braintrust Define Eval Release Gate ↗ Translate completed, intact evaluation evidence into owned ship-or-hold rules. Separates improvement targets from non-negotiable guardrails; does not implement or activate CI.](/skills/braintrustdata-braintrust-define-eval-release-gate/)[### Agent Platform Eval Flywheel ↗ Evaluates and iterates on Google Cloud Agent Platform models and agents, with a confirmation card before every billed evaluation run.](/skills/google-agent-platform-eval-flywheel/)

 [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/)

## Continue your investigation

 - [Choose a toolkit](/tools/)
- [Inspect execution hosts](/agents/)
- [Check tool access](/mcp-servers/)
- [Check project dependencies](/check-repo/)
