--- title: "Arize Evaluator: review, setup & limitations · Undominated.ai" canonical: https://undominated.ai/skills/arize-ai-arize-evaluator/ description: "Creates and runs Arize LLM-as-judge and code evaluators through the ax CLI, with a rule against fabricating scores." --- # Arize Evaluator: review, setup & limitations · Undominated.ai > Creates and runs Arize LLM-as-judge and code evaluators through the ax CLI, with a rule against fabricating scores. [← Explore all skills](/skills/) MODEL EVALUATION / Arize AI # Arize Evaluator Creates and runs Arize LLM-as-judge and code evaluators through the ax CLI, with a rule against fabricating scores. [See setup guidance ↓](#setup)[Original source ↗](https://github.com/Arize-ai/arize-skills/tree/f92b15a001790a4894f5b17e34d3f8f3c5e13514/skills/arize-evaluator) SOURCE REVIEW Reviewed 2026-10-13 Evidence 5 linked sources Publisher Arize AI Licence [MIT ↗](https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/LICENSE) Revision f92b15a00179 [Read what was—and wasn’t—checked ↓](#review) “Handles LLM-as-judge and code evaluator workflows on Arize including creating/updating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, code evaluator, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.” [Arize AI · upstream description ↗](https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/skills/arize-evaluator/SKILL.md) Our analysis follows below. 01 / THE REASONING ## Why this made the selection. - If an evaluation task fails, is cancelled or produces no scores, the file says to report the failure and not run a manual evaluation, invent scores or estimate percentages. - It recommends a backfill of about 100 historical spans before continuous monitoring, and says never to read .env files or ask the user to paste secrets into chat. ### A good fit for - Setting up an LLM-as-judge evaluator on Arize spans or experiments - Diagnosing a task run that scored zero spans ### Weigh up before choosing - The file tells the agent to run ax commands directly without checking versions, environment variables or profiles first. The four reference files it links were not opened. - Judge calls use provider credentials stored as Arize AI integrations, so the judge model and its cost depend on that integration. The install commands do not pin this revision. - A failed or cancelled evaluation must be reported as a failure; the source explicitly prohibits substituting an agent-written manual evaluation for Arize output. 02 / THE REVIEW RECORD ## What we actually inspected. Source review has boundaries. A clear record is more useful than a “safe” badge. ### Material inspected - skills/arize-evaluator/SKILL.md (prerequisites, concepts, workflows, template rules, troubleshooting) - README.md (installation, prerequisites) - LICENSE MIT, copyright 2025 Arize AI, Inc. ### Our findings - Labels in --classification-choices must match the template exactly; omitting them fails runs with a missing-rails error, per the file. - Filters must use attributes.openinference.span.kind; span_kind silently matches nothing, per the file. ### Not established by this review - The installer was not run, and the skill was not executed against a project or service. No ax command was run and no Arize space was contacted. - The references/ files were not opened. - Not executed: it was not installed or run. The review applies to the material and revision named here. A newer upstream release can change its behavior. 03 / PUT IT TO WORK ## Add a skill to your workflow. [Upstream setup instructions ↗](https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/README.md) DOCUMENTED COMMAND npx skills add Arize-ai/arize-skills Copy command ↗ Run in your terminal to choose your AI tools, project or global scope, and installation method. This short command follows the current upstream version; the review covers the revision shown above. - The README's interactive form is npx skills add Arize-ai/arize-skills; its non-interactive form installs every skill with --skill "*" --yes. Choose arize-evaluator in the wizard. - Neither command pins revision f92b15a001790a4894f5b17e34d3f8f3c5e13514. Compare the installed SKILL.md and references with that revision. - Configure credentials with an ax profile; the file says never to ask for a key in chat. ### Before you start - Node.js and npm for the documented Skills installer - The ax CLI (arize-ax-cli) and an Arize space with an AI integration ### Compatibility Agent Skills-compatible hosts · Cursor · Claude Code · Codex · GitHub Copilot · Windsurf · Arize AX via the ax CLI ### Host-mediated Arize evaluator and task writes - Creates and updates evaluators and tasks, and triggers evaluation runs, in an Arize space through ax. - Judge runs call an LLM provider through credentials stored as Arize AI integrations. ### Cost model Source is MIT. Arize usage and judge-model calls are billed by their providers. Those charges were not measured. 04 / FOLLOW THE EVIDENCE ## The source trail. Our notes are separate from the original resource. Check upstream before adopting a new version. - [Skill definition ↗](https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/skills/arize-evaluator/SKILL.md) Checked 2026-10-13 https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/skills/arize-evaluator/SKILL.md Supports: summary, upstreamDescription, whySelected, limitations, access, review - [Install instructions and skill index ↗](https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/README.md) Checked 2026-10-13 https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/README.md Supports: install, compatibility - [Repository licence ↗](https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/LICENSE) Checked 2026-10-13 https://github.com/Arize-ai/arize-skills/blob/f92b15a001790a4894f5b17e34d3f8f3c5e13514/LICENSE Supports: license - [2026-10-15 W23 source re-review at the existing pin ↗](https://raw.githubusercontent.com/Arize-ai/arize-skills/f92b15a001790a4894f5b17e34d3f8f3c5e13514/skills/arize-evaluator/SKILL.md) Checked 2026-10-15 https://raw.githubusercontent.com/Arize-ai/arize-skills/f92b15a001790a4894f5b17e34d3f8f3c5e13514/skills/arize-evaluator/SKILL.md Supports: upstreamDescription, review.notTested, limitations - [2026-10-15 W23 source re-review at the existing pin ↗](https://raw.githubusercontent.com/Arize-ai/arize-skills/f92b15a001790a4894f5b17e34d3f8f3c5e13514/README.md) Checked 2026-10-15 https://raw.githubusercontent.com/Arize-ai/arize-skills/f92b15a001790a4894f5b17e34d3f8f3c5e13514/README.md Supports: install.command KEEP COMPARING ## Other approaches to consider. Related by category or shared topics. These are alternatives to inspect, not a measured quality order. [### Agent Platform Eval Flywheel ↗ Evaluates and iterates on Google Cloud Agent Platform models and agents, with a confirmation card before every billed evaluation run.](/skills/google-agent-platform-eval-flywheel/)[### LangSmith Evaluator ↗ Builds LangSmith offline and online evaluators, with rules for name-based deletes and judge model configuration.](/skills/langchain-ai-langsmith-evaluator/)[### Validate an Eval Scorer ↗ Checks an LLM judge or scorer against expert-reviewed labels, using agreement with uncertainty and severity-weighted errors.](/skills/braintrustdata-braintrust-validate-eval-scorer/) [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/) ## Continue your investigation - [Choose a toolkit](/tools/) - [Inspect execution hosts](/agents/) - [Check tool access](/mcp-servers/) - [Check project dependencies](/check-repo/)