MODEL EVALUATION / MLflow
Build a Scorer
Turns user-defined failure criteria into a small scorer implementation and validation plan, then hands registration and evaluation runs to a separate workflow.
“Help the user choose, create, iterate, or align MLflow judges and scorers for an agent, RAG app, LLM app, or GenAI workflow, including MemAlign and domain-expert labels. Build a small set of atomic quality criteria with the cheapest reliable implementation.”
01 / THE REASONING
Why this made the selection.
- Routes each atomic criterion to code, a matching built-in or an LLM judge, with an explicit why-not-cheaper justification and at least one user-outcome check.
- Its alignment reference requires paired expert assessments, a separately labeled holdout and versioned judges rather than presenting an aligned judge's scores as proof the agent improved.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- build-a-scorer/SKILL.md in full, including phases 5–8 and handoffs
- build-a-scorer/references/judge-alignment.md in full
- agent-evaluation/references/scorers.md in full
- README.md
- LICENSE
- Current MLflow GenAI alignment documentation through Context7, supplementary to the pinned skill
Our findings
- Phase 6 asks the user to confirm a table of name, rule/outcome layer, implementation, why-not-cheaper and output before implementation. Phase 7 supplies runnable code and the data contract; Phase 8 stops at a validation plan and handoff.
- The alignment reference requires valid judge assessments as well as completed agent traces, exactly matching judge/label-schema names, and a separate human-labeled holdout. trace.state OK alone does not prove successful scoring.
- align() changes the judge, not the agent. The reference retains the base version until held-out agreement is demonstrated and forbids mixing base/aligned judge versions in a before/after agent comparison.
- MemAlign reflection/embedding settings must be verified against the installed optimizer and recorded with the run. Provider examples and the README's MLflow 3.8+ statement are upstream guidance, not a compatibility test here.
- This replaces the earlier review's statement that the alignment reference and later phases were unopened; no execution or installation claim was added.
Not established by this review
- No MLflow import/introspection, scorer construction, evaluation, registration, alignment, Review App session or model call was run.
- MLflow/provider SDK implementations, optional hooks, scorer-constraints/troubleshooting references and remaining sibling skills were not audited; held-out agreement and actual cost remain unknown.
- Not executed: it was not installed or run.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Add a skill to your workflow.
DOCUMENTED COMMAND
npx skills add mlflow/skills Run in your terminal to choose your AI tools, project or global scope, and installation method. This short command follows the current upstream version; the review covers the revision shown above.
- The README documents npx skills add mlflow/skills, which offers every MLflow skill.
- The command does not pin revision 51a42ea2b07fadb45fb34c7cecf0c87edbe05686. Compare the installed SKILL.md and reference with that revision.
Before you start
- Node.js and npm for the documented Skills installer
- MLflow with GenAI evaluation, and uv if the project uses it
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/SKILL.md
Supports: summary, upstreamDescription, whySelected, bestFor, review, limitations, access
https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/references/judge-alignment.md
Supports: review, limitations, access
https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/agent-evaluation/references/scorers.md
Supports: review, limitations
- Install and requirements ↗Checked
https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/README.md
Supports: install, compatibility, limitations
- Repository licence ↗Checked
https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/LICENSE
Supports: license
https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/alignment
Supports: review
https://raw.githubusercontent.com/mlflow/skills/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/SKILL.md
Supports: limitations, review.notTested