← Explore all skills

MODEL EVALUATION / MLflow

Build a Scorer

Turns user-defined failure criteria into a small scorer implementation and validation plan, then hands registration and evaluation runs to a separate workflow.

“Help the user choose, create, iterate, or align MLflow judges and scorers for an agent, RAG app, LLM app, or GenAI workflow, including MemAlign and domain-expert labels. Build a small set of atomic quality criteria with the cheapest reliable implementation.”

01 / THE REASONING

Why this made the selection.

  • Routes each atomic criterion to code, a matching built-in or an LLM judge, with an explicit why-not-cheaper justification and at least one user-outcome check.
  • Its alignment reference requires paired expert assessments, a separately labeled holdout and versioned judges rather than presenting an aligned judge's scores as proof the agent improved.

02 / THE REVIEW RECORD

What we actually inspected.

Source review has boundaries.
A clear record is more useful than a “safe” badge.

Material inspected

  • build-a-scorer/SKILL.md in full, including phases 5–8 and handoffs
  • build-a-scorer/references/judge-alignment.md in full
  • agent-evaluation/references/scorers.md in full
  • README.md
  • LICENSE
  • Current MLflow GenAI alignment documentation through Context7, supplementary to the pinned skill

Our findings

  • Phase 6 asks the user to confirm a table of name, rule/outcome layer, implementation, why-not-cheaper and output before implementation. Phase 7 supplies runnable code and the data contract; Phase 8 stops at a validation plan and handoff.
  • The alignment reference requires valid judge assessments as well as completed agent traces, exactly matching judge/label-schema names, and a separate human-labeled holdout. trace.state OK alone does not prove successful scoring.
  • align() changes the judge, not the agent. The reference retains the base version until held-out agreement is demonstrated and forbids mixing base/aligned judge versions in a before/after agent comparison.
  • MemAlign reflection/embedding settings must be verified against the installed optimizer and recorded with the run. Provider examples and the README's MLflow 3.8+ statement are upstream guidance, not a compatibility test here.
  • This replaces the earlier review's statement that the alignment reference and later phases were unopened; no execution or installation claim was added.

Not established by this review

  • No MLflow import/introspection, scorer construction, evaluation, registration, alignment, Review App session or model call was run.
  • MLflow/provider SDK implementations, optional hooks, scorer-constraints/troubleshooting references and remaining sibling skills were not audited; held-out agreement and actual cost remain unknown.
  • Not executed: it was not installed or run.

The review applies to the material and revision named here. A newer upstream release can change its behavior.

03 / PUT IT TO WORK

Add a skill to your workflow.

Upstream setup instructions ↗

DOCUMENTED COMMAND

npx skills add mlflow/skills

Run in your terminal to choose your AI tools, project or global scope, and installation method. This short command follows the current upstream version; the review covers the revision shown above.

  1. The README documents npx skills add mlflow/skills, which offers every MLflow skill.
  2. The command does not pin revision 51a42ea2b07fadb45fb34c7cecf0c87edbe05686. Compare the installed SKILL.md and reference with that revision.

Before you start

  • Node.js and npm for the documented Skills installer
  • MLflow with GenAI evaluation, and uv if the project uses it

04 / FOLLOW THE EVIDENCE

The source trail.

Our notes are separate from the original resource.
Check upstream before adopting a new version.

  1. https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/SKILL.md

    Supports: summary, upstreamDescription, whySelected, bestFor, review, limitations, access

  2. https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/references/judge-alignment.md

    Supports: review, limitations, access

  3. https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/agent-evaluation/references/scorers.md

    Supports: review, limitations

  4. https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/README.md

    Supports: install, compatibility, limitations

  5. Repository licence ↗Checked

    https://github.com/mlflow/skills/blob/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/LICENSE

    Supports: license

  6. https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/alignment

    Supports: review

  7. https://raw.githubusercontent.com/mlflow/skills/51a42ea2b07fadb45fb34c7cecf0c87edbe05686/build-a-scorer/SKILL.md

    Supports: limitations, review.notTested

Evidence & Ask