MODEL EVALUATION / Arize AI
Arize Evaluator
Creates and runs Arize LLM-as-judge and code evaluators through the ax CLI, with a rule against fabricating scores.
“Handles LLM-as-judge and code evaluator workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, code evaluator, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.”
01 / THE REASONING
Why this made the selection.
- If an evaluation task fails, is cancelled or produces no scores, the file says to report the failure and not run a manual evaluation, invent scores or estimate percentages.
- It recommends a backfill of about 100 historical spans before continuous monitoring, and says never to read .env files or ask the user to paste secrets into chat.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- skills/arize-evaluator/SKILL.md (prerequisites, concepts, workflows, template rules, troubleshooting)
- README.md (installation, prerequisites)
- LICENSE MIT, copyright 2025 Arize AI, Inc.
Our findings
- Labels in --classification-choices must match the template exactly; omitting them fails runs with a missing-rails error, per the file.
- Filters must use attributes.openinference.span.kind; span_kind silently matches nothing, per the file.
Not established by this review
- The installer was not run, and the skill was not executed against a project or service. No ax command was run and no Arize space was contacted.
- The references/ files were not opened.
- Not executed: it was not installed or run.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Add a skill to your workflow.
DOCUMENTED COMMAND
npx skills add Arize-ai/arize-skillsRun in your terminal to choose your AI tools, project or global scope, and installation method. This short command follows the current upstream version; the review covers the revision shown above.
- The README's interactive form is npx skills add Arize-ai/arize-skills; its non-interactive form installs every skill with --skill "*" --yes. Choose arize-evaluator in the wizard.
- Neither command pins revision f92b15a001790a4894f5b17e34d3f8f3c5e13514. Compare the installed SKILL.md and references with that revision.
- Configure credentials with an ax profile; the file says never to ask for a key in chat.
Before you start
- Node.js and npm for the documented Skills installer
- The ax CLI (arize-ax-cli) and an Arize space with an AI integration
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
- Skill definition ↗Checked
https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/skills/arize-evaluator/SKILL.md
Supports: summary, upstreamDescription, whySelected, limitations, access, review
https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/README.md
Supports: install, compatibility
- Repository licence ↗Checked
https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/LICENSE
Supports: license
https://raw.githubusercontent.com/Arize-ai/arize-skills/f92b15a001790a4894f5b17e34d3f8f3c5e13514/skills/arize-evaluator/SKILL.md
Supports: upstreamDescription, review.notTested, limitations
https://raw.githubusercontent.com/Arize-ai/arize-skills/f92b15a001790a4894f5b17e34d3f8f3c5e13514/README.md
Supports: install.command