MODEL EVALUATION / Google
Agent Platform Eval Flywheel
Evaluates and iterates on Google Cloud Agent Platform models and agents, with a confirmation card before every billed evaluation run.
“Measures and improves the quality of AI models and agents on Google Cloud using the Eval Quality Flywheel methodology. Use when generating synthetic user scenarios, evaluating an agent or model, building an eval dataset, picking or writing evaluation metrics, analyzing failures, comparing results before and after a fix, or when guidance is needed on Agent Platform eval methodology — including dataset schema, LLM-as-judge scoring, and common failure causes. For fine-tuning, use agent-platform-tuning. For general production deployment, use agent-platform-deploy.”
01 / THE REASONING
Why this made the selection.
- Evaluation, inference and scenario-generation calls need their own Yes/No confirmation for every run, asked in one turn and executed only after the reply.
- The file says never to fabricate scores or claim results not read from an actual result object, and to confirm all cases pass before declaring success.
02 / THE REVIEW RECORD
What we actually inspected.
Source review has boundaries.
A clear record is more useful than a “safe” badge.
Material inspected
- skills/cloud/agent-platform-eval-flywheel/SKILL.md (frontmatter, when to use, safety and confirmation tiers, setup, failure analysis, iteration, proving your work, rules of engagement)
- README.md (installation, skill list, plugins, licence)
- LICENSE Apache-2.0 text
- skills/cloud/agent-platform-eval-flywheel/SKILL.md
- README.md
- LICENSE
Our findings
- Helper scripts that only inspect, validate, parse or compare results are classed as read-only and need no confirmation.
- After a fix, compare_results.py must show the target metric improved and no other metric regressed.
- The evaluation instructions distinguish prompt-only datasets from datasets containing user-supplied responses, warning that another inference pass overwrites those responses. They require saving both JSON and HTML results.
Not established by this review
- The installer was not run, and the skill was not executed against a project or service. No evaluation was run and no Google Cloud project was contacted.
- The references/ files and scripts/ were not opened.
- Not executed: it was not installed or run.
The review applies to the material and revision named here. A newer upstream release can change its behavior.
03 / PUT IT TO WORK
Add a skill to your workflow.
DOCUMENTED COMMAND
npx skills add google/skillsRun in your terminal to choose your AI tools, project or global scope, and installation method. This short command follows the current upstream version; the review covers the revision shown above.
- The README documents npx skills add google/skills and says to select specific skills, target agents and scope when prompted. Select agent-platform-eval-flywheel.
- The command does not pin revision 4601752b28d601d54efb39b06a37488107fc6b1f. Compare the installed directory with that revision.
- Approve each evaluation confirmation card only after checking project, location, model and dataset.
Before you start
- Node.js and npm for the documented Skills installer
- A Google Cloud project with Agent Platform access, GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION, and Python for the scripts
04 / FOLLOW THE EVIDENCE
The source trail.
Our notes are separate from the original resource.
Check upstream before adopting a new version.
- Skill definition ↗Checked
https://github.com/google/skills/blob/4601752b28d601d54efb39b06a37488107fc6b1f/skills/cloud/agent-platform-eval-flywheel/SKILL.md
Supports: summary, upstreamDescription, whySelected, limitations, access, review
https://github.com/google/skills/blob/4601752b28d601d54efb39b06a37488107fc6b1f/README.md
Supports: install, compatibility
- Repository licence ↗Checked
https://github.com/google/skills/blob/4601752b28d601d54efb39b06a37488107fc6b1f/LICENSE
Supports: license
https://raw.githubusercontent.com/google/skills/4601752b28d601d54efb39b06a37488107fc6b1f/skills/cloud/agent-platform-eval-flywheel/SKILL.md
Supports: review.findings, bestFor