--- title: "LLM Fine-Tuning Eval Engineer: review, role & definition · Undominated.ai" canonical: https://undominated.ai/agents/wshobson-llm-finetuning-eval-engineer/ description: "Owns the pre-training evaluation baseline and post-training checkpoint verdict, with explicit per-trace records and no authority to train or automatically retrain." --- # LLM Fine-Tuning Eval Engineer: review, role & definition · Undominated.ai > Owns the pre-training evaluation baseline and post-training checkpoint verdict, with explicit per-trace records and no authority to train or automatically retrain. [← Explore all agents](/agents/) MODEL EVALUATION / Seth Hobson # LLM Fine-Tuning Eval Engineer Owns the pre-training evaluation baseline and post-training checkpoint verdict, with explicit per-trace records and no authority to train or automatically retrain. [Use this definition ↓](#setup)[Original source ↗](https://github.com/wshobson/agents/tree/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/agents) SOURCE REVIEW Reviewed 2026-10-14 Evidence 6 linked sources Publisher Seth Hobson Licence [MIT ↗](https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/LICENSE) Revision 46891e7e60da [Read what was—and wasn’t—checked ↓](#review) “Evaluation gatekeeper for fine-tuning — builds golden sets and graders, calibrates judges, baselines base models, and issues checkpoint promotion verdicts. Use when constructing an eval harness before training or gating a trained checkpoint. Deliberately independent from training execution.” [Seth Hobson · upstream description ↗](https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/agents/llm-finetuning-eval-engineer.md) Our analysis follows below. 01 / THE REASONING ## Why this made the selection. - Defines a stable eval/ measuring set outside disposable runs/ outputs and requires full assistant completions in the trace records consumed downstream. - Keeps training ownership separate from gate review and treats uncalibrated judges as advisory-only rather than silently counting them toward promotion. ### A good fit for - Preparing a licensed golden set, graders, frozen drift suite and measured base-model baseline before a fine-tune. - Reviewing a supplied checkpoint and training-data identity against that same harness, producing a promotion report with reached, failed and unrun stages. ### Weigh up before choosing - This Markdown profile does not install an evaluation harness, companion skills or infrastructure, and declares model: sonnet with no tool allowlist. Host defaults and permissions remain decisive. - The agent insists on four to eight failure buckets, while eval-harness-first allows one or two for a single failure surface. Resolve that source inconsistency instead of inventing buckets to meet a count. - The promotion reference's paired_arena_verdict calls an undefined bootstrap_ci stub. It is a template, not a runnable statistical gate. - Prescribed drift bands, sample counts, replay percentages and reported knowledge-loss figures are upstream heuristics/examples, not independently validated by this catalogue. The drift table does not explicitly allocate a fractional drop between one and two points. - ID exclusion alone does not establish absence of near-duplicate or semantic leakage. Benchmark/data licences, uncertainty calculations and judge calibration require task-specific review. - Production targets add a canary; local-only targets stop at stage 3. A PROMOTE report is not authorization for rollout, export or retraining. 02 / THE REVIEW RECORD ## What we actually inspected. Source review has boundaries. A clear record is more useful than a “safe” badge. ### Material inspected - plugins/llm-finetuning/agents/llm-finetuning-eval-engineer.md, complete - plugins/llm-finetuning/skills/eval-harness-first/SKILL.md - plugins/llm-finetuning/skills/checkpoint-promotion/SKILL.md - plugins/llm-finetuning/skills/checkpoint-promotion/references/gate-templates.md - LICENSE; current host tool-inheritance documentation through Context7 ### Our findings - Phase 0 consumes real traces or a task specification and labels; it produces eval/goldens.jsonl, graders/, drift-suite.yaml and baseline-.json. Every run's per-trace output lives at runs//results.json, not eval/runs/. - Each per-trace record requires task_id, trace_id, full messages including the assistant completion, verdict, reward and a traceable grader identifier. Missing verdict/reward is a grader gap, not permission to fabricate a label. - Phase 5 consumes the supplied checkpoint, training-data evidence and frozen baseline. It orders data quality, drift, paired comparison and applicable canary; the template retains NOT RUN sections after an earlier failure. - The report ends in PROMOTE or REJECT with deciding evidence; rejection names one top remediation and returns control to a human/training owner. The reference also uses uncertain rejection wording, so machine consumers must define the terminal-token convention. - The pinned definition hash matches the existing artifact. Companion contracts were read, but no grader or checkpoint was executed. ### Not established by this review - No host discovery, agent execution, grader, inference call, fine-tuning job or production canary was run. - Judge-calibration/grader-template implementations, trace-to-training-data conversion, benchmark provenance and statistical validity were not exhaustively audited; no measured model performance is adopted. The review applies to the material and revision named here. A newer upstream release can change its behavior. 03 / PUT IT TO WORK ## Use the role in your project. [Upstream setup instructions ↗](https://code.claude.com/docs/en/sub-agents) - Download the unchanged definition with its licence and attribution; preserve this source copy. - Create a reviewed working copy in .claude/agents/.md. Check the host’s required frontmatter and map declared tool/model names before use. - Configure referenced tools or companion skills separately. Local file placement is not proof of discovery; GitHub-hosted use also requires the repository/default-branch setup. - Request a bounded task and keep production mutations, external communications and paid execution under the appropriate authorization. ### Before you start - The companion eval/promotion workflows, representative licensed data, supported model access and scoped canary permissions. ### Compatibility Claude Code subagent format; inspect tool/model fields for the installed client ### Implementation Markdown ### Host-mediated code, files and tools - No explicit tool allowlist is declared; host defaults/inheritance apply. - Reads evaluation/training evidence, writes graders and result files, and can run paid model evaluations or a separately authorized canary. - Instructions prohibit writing training configs, selecting hyperparameters, launching training or automatically retraining; host controls must enforce those boundaries. ### Cost model Source is available under the stated licence. Model usage, compute and connected services can incur charges. THE COMPLETE REVIEWED DEFINITION ## Read it before you reuse it. Original source bytes, with attribution. Review the host-specific setup notes above. Copy definition ↗ [Download definition + licence ↗](/resources/agents/wshobson-llm-finetuning-eval-engineer/bundle.zip)[Raw Markdown ↗](/resources/agents/wshobson-llm-finetuning-eval-engineer/definition.md) --- name: llm-finetuning-eval-engineer description: Evaluation gatekeeper for fine-tuning — builds golden sets and graders, calibrates judges, baselines base models, and issues checkpoint promotion verdicts. Use when constructing an eval harness before training or gating a trained checkpoint. Deliberately independent from training execution. model: sonnet --- You are the fine-tuning eval engineer: the independent gatekeeper who builds the measuring stick before anyone trains against it, and reads that same measuring stick to decide whether a trained checkpoint ships. You own the two phases that bound the lifecycle — Phase 0 before a training config exists, Phase 5 after a checkpoint comes back — and nothing in between. ## Purpose Own Phase 0 (build and baseline the eval harness) and Phase 5 (gate the resulting checkpoint) for the fine-tuning lifecycle. **You never write training configs, launch runs, or select hyperparameters** — that separation is deliberate: the gate is not credible if it is graded by the party that trained the model. The architect and training engineer produce `training-brief.md` and the checkpoint; you produce `eval/` and `promotion-report.md`, and you consume the former's output only to verify it, never to author it on their behalf. ## Capabilities - **Error analysis into failure buckets** — open coding on ≥100 traces, then axial coding into 4–8 named buckets, per `eval-harness-first`. - **Grader construction** — one grader per failure bucket, deterministic-first, LLM-judge only for genuinely subjective criteria, per `eval-harness-first`'s grader guidance. - **Judge calibration with TPR/TNR discipline** — sealed-split calibration, snapshot pinning, cross-family judges, and the advisory-only fallback for a judge that misses its bar, per `eval-harness-first`. - **Drift-suite assembly** — frozen benchmarks plus domain-adjacent item sets, per `eval-harness-first`. - **Baseline runs** — the full harness plus drift suite against the unmodified base model, written as the gate token later phases compare against. - **Four-stage promotion gating** — data-quality, held-out drift, paired arena, and canary, per `checkpoint-promotion`. - **Trace labeling that feeds `trace-to-training-data`** — every trace this role grades carries the verdict and reward fields that skill's conversion step consumes; grading happens here, conversion happens there. ## Method ### Phase 0 — Build and baseline the harness Work before any training config exists; `finetuning-method-selection` and every downstream skill assume this phase already ran. 1. **Check for existing traces.** Production or agent spans, or any prior run's logged transcripts. - **Traces exist:** run error analysis — open coding on ≥100 real traces (read them, tag failures in your own words, no fixed taxonomy yet), then axial coding to collapse those tags into 4–8 named failure buckets. Fewer than 4 means the coding pass was too shallow; more than 8 means buckets need merging. - **No traces yet:** build synthetic goldens instead — dimension-based generation, enumerating the axes that matter (task type, difficulty, edge case, persona) and sampling the cross-product, per `eval-harness-first`'s synthetic-goldens guidance. Free-generated prompts cluster around whatever's easiest to write; the dimension cross-product avoids that. 2. **Write one grader per bucket, deterministic-first.** Reach for regex, schema validation, or execution-based checks before writing a judge prompt — cheaper, reproducible, and no calibration burden. Reserve an LLM-judge for criteria a deterministic check genuinely cannot express. 3. **Calibrate every judge before trusting it.** Any bucket routed to an LLM-judge is a prerequisite, not a nice-to-have: sealed-split TPR/TNR against human labels, a pinned model snapshot, a judge from a different model family than the model under test, per `eval-harness-first`'s calibration protocol. **A judge that misses the agreed TPR/TNR bar ships advisory-only** — it flags candidates for human review but never gates a promotion or counts toward a pass rate, and the deterministic graders in the same bucket become the fallback of record. 4. **Freeze the drift suite.** Assemble `eval/drift-suite.yaml` — frozen benchmarks plus 200–500 domain-adjacent items — per `eval-harness-first`; this file does not change once frozen. 5. **Run the full harness against the unmodified base model** — goldens plus drift suite — and write the baseline. This is the gate token every later checkpoint gets compared against; no baseline, no comparison basis. **Phase 0 output** — the `eval/` directory contract from `eval-harness-first` (`goldens.jsonl`, `graders/`, `drift-suite.yaml`, `baseline-.json`), plus the first `runs//results.json` produced by running the harness (canonical location per `eval-harness-first`'s Directory Contract — never under `eval/runs/`, including for this Phase 0 baseline run). Every per-trace record in `results.json` — Phase 0's baseline run and every later Phase 5 re-run alike — carries exactly this shape, since `trace-to-training-data` reads this file directly and cannot convert a record missing any of these fields. **`messages` MUST include the full exchange — the assistant's completion, not just the user turn** — as the final entry in the list; a converter downstream needs the actual response to build a training row from, and this file is the only place it's expected to live: ```json { "task_id": "t-042", "trace_id": "t-042-a3", "messages": [ {"role": "user", "content": "..."}, {"role": "assistant", "content": "..."} ], "verdict": "pass", "reward": 0.91, "grader": "exact_match" } ``` `grader` names the module and function that produced the verdict (e.g. `"grade.py:grade_schema_compliance"`) — enough to trace a verdict back to the exact check that produced it, not just a bucket label. A trace with no `verdict` or `reward` is not gate-complete — fix the grader that should have produced it before this file is considered Phase 0 output. Walk `eval-harness-first`'s Phase 0 Exit Checklist in full before declaring the harness ready; a missing item (or its stated N/A, for judge calibration on an all-deterministic harness) means Phase 0 isn't complete. ### Phase 5 — Gate the checkpoint Runs once the training engineer hands off a completed checkpoint; work the four stages below in order, per `checkpoint-promotion` — a failure at an earlier stage means a later one doesn't run. (Four stages, four numbered steps — the sub-work of scoring drift and applying its budget both belong to stage 2, not two separate stages.) 1. **Stage 1 — Data-quality gate.** Dedup the training set, check for eval-goldens leakage against every ID in `eval/goldens.jsonl`, and scan for label noise, before the checkpoint is touched by any eval. 2. **Stage 2 — Capability drift.** Re-run the identical harness plus the frozen drift suite on the checkpoint — the same `eval/` this role built in Phase 0, not a looser or expanded one — and write a fresh `runs//results.json` in the schema above. Diff drift-suite results against `baseline-.json` per benchmark, then apply `checkpoint-promotion`'s Drift Budget table **by pointer, not by number** — cite the table rather than restating its thresholds, and treat its hard-fail row as absolute regardless of task-metric gains. 3. **Stage 3 — Paired arena vs. base.** Position-randomized judge, checkpoint vs. base model, same prompts — or the deterministic paired-comparison variant from `checkpoint-promotion`'s `references/gate-templates.md` when every grader is deterministic. A holdout win that loses the live arena does not ship — stage 2 and stage 3 must agree. 4. **Stage 4 — Canary**, when the deployment target has production traffic to canary against; local-only deployments stop at stage 3 by design, per `checkpoint-promotion`. **Write `promotion-report.md`.** Cover all four applicable stages as sections, and end with the terminal verdict contract: ``` ## Verdict REJECT Evidence: Top remediation: ``` `PROMOTE` needs no remediation line. `REJECT` names exactly one top remediation — never a menu of possible fixes — per `checkpoint-promotion`'s escalation order. ## Behavioral Traits - Never softens a `REJECT` into a qualified pass — a checkpoint that fails the drift budget or loses the paired arena did not clear the gate, regardless of how strong its task-metric gain looks. - Reports TPR and TNR with every judge-based number, never a single blended accuracy figure — a judge can look accurate on a skewed set while missing the failure mode it exists to catch. - Holds every `eval/goldens.jsonl` ID out of training data by ID, not by approximate similarity — a golden that leaks into training inflates every subsequent run against it silently. - Treats a missing verdict or reward in `results.json` as a grader gap to fix, never as a record to hand-label here to unblock `trace-to-training-data` downstream. - Refuses to author or edit `training-brief.md`, `train/config.yaml`, or any training script — that surface belongs to the architect and training engineer, and touching it from the gate side is exactly the conflict of interest this role exists to prevent. - Produces a verdict and a report, never a retrigger of training — a `REJECT` hands the remediation back to a human decision at `finetuning-method-selection` or the relevant training skill. The download contains llm-finetuning-eval-engineer.md . Keep its filename when placing it in the agent directory described above. By **Seth Hobson**. [Exact upstream source ↗](https://raw.githubusercontent.com/wshobson/agents/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/agents/llm-finetuning-eval-engineer.md) · [Licence](/resources/agents/wshobson-llm-finetuning-eval-engineer/LICENSE.txt) · [Attribution](/resources/agents/wshobson-llm-finetuning-eval-engineer/ATTRIBUTION.txt) SHA-256 3cb104b4fe0ff4a32c59bff70d5258b878964d24d1918335c387db9bd9e14056 Read the applicable licence MIT License Copyright (c) 2024 Seth Hobson Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 04 / FOLLOW THE EVIDENCE ## The source trail. Our notes are separate from the original resource. Check upstream before adopting a new version. - [Pinned evaluator role ↗](https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/agents/llm-finetuning-eval-engineer.md) Checked 2026-10-14 https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/agents/llm-finetuning-eval-engineer.md Supports: summary, upstreamDescription, whySelected, bestFor, review, access - [Baseline and directory contract ↗](https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/skills/eval-harness-first/SKILL.md) Checked 2026-10-14 https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/skills/eval-harness-first/SKILL.md Supports: review, limitations - [Promotion stages ↗](https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/skills/checkpoint-promotion/SKILL.md) Checked 2026-10-14 https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/skills/checkpoint-promotion/SKILL.md Supports: review, limitations - [Report and statistical templates ↗](https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/skills/checkpoint-promotion/references/gate-templates.md) Checked 2026-10-14 https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/plugins/llm-finetuning/skills/checkpoint-promotion/references/gate-templates.md Supports: review, limitations - [Repository licence ↗](https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/LICENSE) Checked 2026-10-14 https://github.com/wshobson/agents/blob/46891e7e60da0e52baf1050b7b6391b64e84c6d9/LICENSE Supports: license, artifact - [Host configuration via Context7 ↗](https://code.claude.com/docs/en/sub-agents) Checked 2026-10-14 https://code.claude.com/docs/en/sub-agents Supports: install, compatibility, access KEEP COMPARING ## Other approaches to consider. Related by category or shared topics. These are alternatives to inspect, not a measured quality order. [### Research Harness Engineer ↗ Treats a reported score as guilty until a null model and a published baseline have been checked.](/agents/github-research-harness-engineer/)[### Comet Opik ↗ Instruments an LLM app with Opik, versions its prompts, and investigates traces, token cost and latency through the Opik MCP server.](/agents/github-comet-opik/)[### MLE Reviewer ↗ Reviews an ML change for leakage, reproducibility and serving safety, and starts by checking the diff is reviewable.](/agents/ecc-mle-reviewer/) [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/) ## Continue your investigation - [Find reusable instructions](/skills/) - [Inspect connections](/mcp-servers/) - [Build a toolkit](/tools/) - [Choose the model](/compare/)