← Explore all skills

AI EVALUATION / Braintrust

Braintrust Design Eval Experiment

Prepare a comparison manifest before inference: freeze the serving path, tool permissions, pairing, treatment implementation and stopping rules. It does not run an experiment.

“Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected.”

01 / THE REASONING

Why this made the selection.

  • Makes batching, provider, region, cache state and decoding part of the controlled comparison instead of treating a model name as a reproducible configuration.
  • Requires per-arm tool manifests and implementable treatments, complementing existing scorer and dataset-size reviews without duplicating them.

02 / THE REVIEW RECORD

What we actually inspected.

Source review has boundaries.
A clear record is more useful than a “safe” badge.

Material inspected

  • Pinned definition and references/experiment-manifest.md
  • Its interaction-contract.md and platform-mechanics.md
  • Root README, LICENSE and recursive Git tree

Our findings

  • The manifest reference records provider, endpoint tier, region, decoding, cache state and batching, and asks for tool names, schemas and permissions per arm.
  • It excludes an arm that cannot receive a treatment from the treatment-effect claim rather than labeling that arm as receiving no benefit.
  • It fixes stopping, exclusion and multiplicity rules before collection, distinguishes exploratory from confirmatory runs, and says to restart or disclose a defect instead of changing configuration mid-comparison.
  • Run-integrity guidance requires a concurrency cap, a whole-matrix quota budget and separately recorded trials. These are requirements for a future runner, not mechanisms in this Markdown.
  • The selected directory contains a definition and three Markdown references, without executable helpers, lifecycle hooks, MCP handlers or an enforcement config. The root manual-copy route is independent of runnable example projects.
  • The common contract distinguishes audit from repair and treats trace content as evidence. Apache-2.0 licence terms were read at the same commit.

Not established by this review

  • No skill installation or host discovery
  • No inference request, paired run, quota probe, experiment creation or deletion
  • No statistical computation or live verification of Braintrust platform claims
  • Not executed: it was not installed or run.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

Add a skill to your workflow.

Upstream setup instructions ↗
  1. Use the publisher's individual-directory copy option with the pinned design skill and its references, preserving licence notices.
  2. Load it in a compatible host for manifest authoring; do not install the repository's separate runnable study examples as a prerequisite.
  3. Keep all account changes and future execution behind task-specific approval. No install command or host configuration was composed for this record.

Before you start

  • Agent Skills-compatible host; Claude Code and Codex CLI are named by the publisher
  • Proposed arms, intended population and available dataset/scorer identifiers
  • No cloud credentials needed for local manifest authoring; authenticated project access only for optional platform changes

04 / FOLLOW THE EVIDENCE

The source trail.

Our notes are separate from the original resource.
Check upstream before adopting a new version.

  1. https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/SKILL.md

    Supports: Exact description excerpt and pre-run scope

  2. https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/references/experiment-manifest.md

    Supports: Batching and serving controls, tool permissions, quotas and treatment implementation

  3. Interaction contract ↗Checked

    https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/references/interaction-contract.md

    Supports: Uncertainty labels, audit boundaries and trace trust boundary

  4. Platform mechanics ↗Checked

    https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/references/platform-mechanics.md

    Supports: Pinning, naming, caching and deletion limitations

  5. https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md

    Supports: Manual skill-directory copy and supported host claims

  6. Pinned licence ↗Checked

    https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE

    Supports: Apache-2.0 licence inspection

  7. https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-design-eval-experiment/SKILL.md

    Supports: upstreamDescription, review.notTested, review.reviewedAt

Evidence & Ask