AI EVALUATION / Braintrust
Braintrust Define Eval Release Gate
Translate completed, intact evaluation evidence into owned ship-or-hold rules. Separates improvement targets from non-negotiable guardrails; does not implement or activate CI.
“Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate failed a gate and what evidence would justify reconsideration. Do not use for general result analysis without a deployment decision.”
01 / THE REASONING
Why this made the selection.
- Consumes a scorer-validation verdict and confirmatory results to produce an explicit decision contract, a different job from constructing data or computing sample size.
- Requires error-accounting completeness, owner-sourced thresholds and separate guardrail constraints, so a better average cannot silently excuse missing rows or safety failures.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- Pinned definition and references/gate-rows.md
- Its interaction-contract.md and platform-mechanics.md
- Root README, LICENSE and recursive Git tree
Our findings
- The definition makes reconciled effective sample counts and per-arm errors a precondition: unexplained missing items cannot be treated as a passing population.
- Each gate row names a threshold source and owner, scorer identity and fitness for gating, and a failure interpretation; unresolved product thresholds remain Needs decision.
- The reference distinguishes all-attempt success from any-attempt success and calls for upper bounds or worst runs where appropriate rather than substituting an average.
- Red-team findings enter as existence constraints re-tested each round; improvement metrics cannot buy down guardrails in a weighted sum.
- Only Markdown and its three references occur in the selected directory. There is no executable CI check, installation hook, MCP handler or credential configuration to enforce the prose.
- Apache-2.0 licence terms were inspected at the definition's commit. The record links to the source and does not redistribute a definition archive.
Not established by this review
- No installation, host activation or CI wiring
- No statistical bound computation, gating decision or deployment
- No Braintrust read, scorer save, promotion block or experiment deletion
- Not executed: it was not installed or run.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Add a skill to your workflow.
- The publisher supports copying a single skill directory with its references into an Agent Skills host; preserve the inspected revision and licence notices.
- Use it to author or audit a gate contract against supplied results and scorer fitness statements.
- Implementation, credential grants and promotion authority remain separate tasks. This packet does not activate a gate or reproduce the publisher's broad installation loop.
Before you start
- Agent Skills-compatible host
- Intact confirmatory evaluation evidence and validated scorer identities
- Product-owner decisions for tolerances; cloud or CI access only if separately implementing the resulting gate
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-define-eval-release-gate/SKILL.md
Supports: Description excerpt, decision scope and error-accounting precondition
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-define-eval-release-gate/references/gate-rows.md
Supports: Illustrative threshold warning, failure interpretation and owner fields
- Interaction contract ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-define-eval-release-gate/references/interaction-contract.md
Supports: Audit versus repair; uncertainty and evidence boundaries
- Platform mechanics ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-define-eval-release-gate/references/platform-mechanics.md
Supports: Custom-calculation needs and destructive cleanup caveat
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/README.md
Supports: Individual-directory install option and compatible hosts
- Pinned licence ↗Checked
https://github.com/braintrustdata/eval-library/blob/04486b39047b5ba230f795e52f2935fef651e789/LICENSE
Supports: Apache-2.0 redistribution conditions
https://raw.githubusercontent.com/braintrustdata/eval-library/04486b39047b5ba230f795e52f2935fef651e789/skills/braintrust-define-eval-release-gate/SKILL.md
Supports: upstreamDescription, review.notTested, review.reviewedAt