---
title: "Build a Scorer: review, setup & limitations · Undominated.ai"
canonical: https://undominated.ai/skills/mlflow-build-a-scorer/
description: "Turns user-defined failure criteria into a small scorer implementation and validation plan, then hands registration and evaluation runs to a separate workflow."
---

# Build a Scorer: review, setup & limitations · Undominated.ai

> Turns user-defined failure criteria into a small scorer implementation and validation plan, then hands registration and evaluation runs to a separate workflow.

[← Explore all skills](/skills/)

MODEL EVALUATION / MLflow

# Build a Scorer

Turns user-defined failure criteria into a small scorer implementation and validation plan, then hands registration and evaluation runs to a separate workflow.

 [See setup guidance ↓](#setup)[Original source ↗](https://github.com/mlflow/skills/tree/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer)

SOURCE REVIEW

 Reviewed 2026-10-14
 Evidence 7 linked sources
 Publisher MLflow
 Licence [Apache-2.0 ↗](https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/LICENSE)
 Revision 51a42ea2b07f
 [Read what was—and wasn’t—checked ↓](#review)

“Help the user choose, create, iterate, or align MLflow judges and scorers for an agent, RAG app, LLM app, or GenAI workflow, including MemAlign and domain-expert labels. Build a small set of atomic quality criteria with the cheapest reliable implementation.”

 [MLflow · upstream description ↗](https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/SKILL.md) Our analysis follows below.

01 / THE REASONING

## Why this made the selection.

 - Routes each atomic criterion to code, a matching built-in or an LLM judge, with an explicit why-not-cheaper justification and at least one user-outcome check.
- Its alignment reference requires paired expert assessments, a separately labeled holdout and versioned judges rather than presenting an aligned judge's scores as proof the agent improved.

### A good fit for

 - Designing scorer code from concrete app failures, expected outcomes and available inputs/outputs/expectations or traces.
- Preparing a handoff containing confirmed criteria, scorer implementation, observable data requirements, expert labels and a held-out validation plan.

### Weigh up before choosing

 - This is scorer design, not an end-to-end evaluation runner. Phase 8 assigns dataset creation, registration, runs and result interpretation to agent-evaluation, with tracing gaps handed to instrumenting-with-mlflow-tracing.
- The source caps the initial suite at three to five checks and treats initial LLM judges as drafts. These are workflow choices, not measured reliability guarantees.
- Installed scorer APIs and required_columns must be checked; source tables, constructor signatures and model defaults are version-sensitive. RETRIEVER-only checks do not become valid for TOOL-only traces.
- The source permits not-applicable rows to be skipped or represented by passing Feedback. Choose and report that denominator policy explicitly rather than silently inflating a pass rate.
- build-a-scorer requires an explicit judge model, but the referenced scorers guide recommends a default in its best-practices section. Resolve that conflict in the workflow; implicit defaults can select an unconfigured or billable provider.
- The README's source-install example clones into skills/ then copies mlflow-skills/*, and its npx command is unpinned. The existing receipt is retained, not certified as a reproducible installation.
- The source says session-level built-in judges cannot use align(), so a contested criterion may need a custom judge before alignment.

02 / THE REVIEW RECORD

## What we actually inspected.

Source review has boundaries. A clear record is more useful than a “safe” badge.

### Material inspected

 - build-a-scorer/SKILL.md in full, including phases 5–8 and handoffs
- build-a-scorer/references/judge-alignment.md in full
- agent-evaluation/references/scorers.md in full
- README.md
- LICENSE
- Current MLflow GenAI alignment documentation through Context7, supplementary to the pinned skill

### Our findings

 - Phase 6 asks the user to confirm a table of name, rule/outcome layer, implementation, why-not-cheaper and output before implementation. Phase 7 supplies runnable code and the data contract; Phase 8 stops at a validation plan and handoff.
- The alignment reference requires valid judge assessments as well as completed agent traces, exactly matching judge/label-schema names, and a separate human-labeled holdout. trace.state OK alone does not prove successful scoring.
- align() changes the judge, not the agent. The reference retains the base version until held-out agreement is demonstrated and forbids mixing base/aligned judge versions in a before/after agent comparison.
- MemAlign reflection/embedding settings must be verified against the installed optimizer and recorded with the run. Provider examples and the README's MLflow 3.8+ statement are upstream guidance, not a compatibility test here.
- This replaces the earlier review's statement that the alignment reference and later phases were unopened; no execution or installation claim was added.

### Not established by this review

 - No MLflow import/introspection, scorer construction, evaluation, registration, alignment, Review App session or model call was run.
- MLflow/provider SDK implementations, optional hooks, scorer-constraints/troubleshooting references and remaining sibling skills were not audited; held-out agreement and actual cost remain unknown.
- Not executed: it was not installed or run.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

## Add a skill to your workflow.

[Upstream setup instructions ↗](https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/README.md)

DOCUMENTED COMMAND

 npx skills add mlflow/skills Copy command ↗

Run in your terminal to choose your AI tools, project or global scope, and installation method. This short command follows the current upstream version; the review covers the revision shown above.

 - The README documents npx skills add mlflow/skills, which offers every MLflow skill.
- The command does not pin revision 51a42ea2b07fadb45fb34c7cecf0c87edbe05686. Compare the installed SKILL.md and reference with that revision.

### Before you start

 - Node.js and npm for the documented Skills installer
- MLflow with GenAI evaluation, and uv if the project uses it

### Compatibility

Agent Skills-compatible hosts · Claude Code · Cursor · Codex CLI · Gemini CLI · OpenCode · MLflow GenAI evaluation

### Implementation

Python

### Host-mediated project files

 - Frontmatter allows Read, Write, Bash, Grep and Glob; the workflow calls for installed-Python introspection and writes scorer code.
- Running judge/alignment code can send trace and label content to configured model/embedding providers; review data rights and destination first.
- Review App sessions, dataset creation, registration and production scoring are separate authorized operations, not actions taken by this review.

### Cost model

Source is Apache-2.0. LLM-judge scorers call models billed by their providers. Those charges were not measured.

04 / FOLLOW THE EVIDENCE

## The source trail.

Our notes are separate from the original resource. Check upstream before adopting a new version.

 - [Complete pinned scorer workflow ↗](https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/SKILL.md) Checked 2026-10-14 https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/SKILL.md Supports: summary, upstreamDescription, whySelected, bestFor, review, limitations, access
- [Expert-label alignment and version boundaries ↗](https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/references/judge-alignment.md) Checked 2026-10-14 https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/references/judge-alignment.md Supports: review, limitations, access
- [Referenced scorer/provider guide ↗](https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/agent-evaluation/references/scorers.md) Checked 2026-10-14 https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/agent-evaluation/references/scorers.md Supports: review, limitations
- [Install and requirements ↗](https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/README.md) Checked 2026-10-14 https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/README.md Supports: install, compatibility, limitations
- [Repository licence ↗](https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/LICENSE) Checked 2026-10-14 https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/LICENSE Supports: license
- [Supplementary current alignment docs via Context7 ↗](https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/alignment) Checked 2026-10-14 https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/alignment Supports: review
- [Pinned source review C2-W03 ↗](https://raw.githubusercontent.com/mlflow/skills/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/SKILL.md) Checked 2026-10-15 https://raw.githubusercontent.com/mlflow/skills/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/SKILL.md Supports: limitations, review.notTested

KEEP COMPARING

## Other approaches to consider.

Related by category or shared topics. These are alternatives to inspect, not a measured quality order.

 [### Agent Platform Eval Flywheel ↗ Evaluates and iterates on Google Cloud Agent Platform models and agents, with a confirmation card before every billed evaluation run.](/skills/google-agent-platform-eval-flywheel/)[### Arize Evaluator ↗ Creates and runs Arize LLM-as-judge and code evaluators through the ax CLI, with a rule against fabricating scores.](/skills/arize-ai-arize-evaluator/)[### LangSmith Evaluator ↗ Builds LangSmith offline and online evaluators, with rules for name-based deletes and judge model configuration.](/skills/langchain-ai-langsmith-evaluator/)

 [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/)

## Continue your investigation

 - [Choose a toolkit](/tools/)
- [Inspect execution hosts](/agents/)
- [Check tool access](/mcp-servers/)
- [Check project dependencies](/check-repo/)
