--- title: "Research Harness Engineer: review, role & definition · Undominated.ai" canonical: https://undominated.ai/agents/github-research-harness-engineer/ description: "Treats a reported score as guilty until a null model and a published baseline have been checked." --- # Research Harness Engineer: review, role & definition · Undominated.ai > Treats a reported score as guilty until a null model and a published baseline have been checked. [← Explore all agents](/agents/) MODEL EVALUATION / GitHub # Research Harness Engineer Treats a reported score as guilty until a null model and a published baseline have been checked. [Use this definition ↓](#setup)[Original source ↗](https://github.com/github/awesome-copilot/tree/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/agents) SOURCE REVIEW Reviewed 2026-10-14 Evidence 2 linked sources Publisher GitHub Licence [MIT ↗](https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/LICENSE) Revision 3a685010a7af [Read what was—and wasn’t—checked ↓](#review) “Research harness engineer for experiment campaigns: builds evaluation harnesses that are hard to fool, then keeps every reported number honest - null models first, calibration/held-out separation, baseline reproduction before improvement claims, paired error bars, and guards verified by deliberate breakage.” [GitHub · upstream description ↗](https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/agents/research-harness-engineer.agent.md) Our analysis follows below. 01 / THE REASONING ## Why this made the selection. - The file requires a null model and a reproduced baseline before any improvement claim. - A new guard must be shown to fail on a deliberately broken input before it counts. ### A good fit for - Provide the experiment question, dataset independence units, published baseline recipe and permitted compute/evaluation budget. - Receive a shared metric entrypoint, null/positive controls, split discipline and persisted paired-comparison results. ### Weigh up before choosing - No tool allowlist or executable harness is included; datasets, metrics and compute must be supplied. - The source tells the agent to commit numbers; preserve repository/user commit authorization rather than treating that sentence as permission. - Flat sweeps have multiple possible causes; the suggested gradient-force check is a hypothesis, not a diagnosis. - Its statement that an interval crossing zero is noise is too categorical: report uncertainty/inconclusive evidence, not proof of no effect. 02 / THE REVIEW RECORD ## What we actually inspected. Source review has boundaries. A clear record is more useful than a “safe” badge. ### Material inspected - agents/research-harness-engineer.agent.md - LICENSE - Definition reread at the preserved revision; scoped workflow and output-contract review on 2026-10-14 ### Our findings - Frontmatter name is Research Harness Engineer and there is no tools key. - The file says to score a constant output and an untrained model before anything else. - It says to reproduce a published baseline before claiming an improvement, and not to improve an unmatched baseline. - It says every new guard must be shown to fail on deliberately broken input before it counts. - It requires guards to fail on deliberately broken inputs and keeps algorithm implementations out of duplicated experiment scripts. - Held-out accesses are budgeted/logged, metric conventions are pinned, and an unmatched published baseline blocks improvement claims. ### Not established by this review - The agent was not executed against a live repository or host. The review applies to the material and revision named here. A newer upstream release can change its behavior. 03 / PUT IT TO WORK ## Use the role in your project. [Upstream setup instructions ↗](https://docs.github.com/en/copilot/reference/custom-agents-configuration) - Download the unchanged definition with its licence and attribution; preserve this source copy. - Create a reviewed working copy in .github/agents/.agent.md. Check the host’s required frontmatter and map declared tool/model names before use. - Configure referenced tools or companion skills separately. Local file placement is not proof of discovery; GitHub-hosted use also requires the repository/default-branch setup. - Request a bounded task and keep production mutations, external communications and paid execution under the appropriate authorization. ### Before you start - The project’s data rights, a reproducible baseline, a leakage-aware split and an authorized execution budget. ### Compatibility GitHub Copilot custom-agent format; inspect tool/model fields for the installed client ### Implementation Markdown ### Local role instructions - No tool allowlist is declared. ### Cost model MIT. The file does not state a hosted-model price. THE COMPLETE REVIEWED DEFINITION ## Read it before you reuse it. Original source bytes, with attribution. Review the host-specific setup notes above. Copy definition ↗ [Download definition + licence ↗](/resources/agents/github-research-harness-engineer/bundle.zip)[Raw Markdown ↗](/resources/agents/github-research-harness-engineer/definition.md) --- description: 'Research harness engineer for experiment campaigns: builds evaluation harnesses that are hard to fool, then keeps every reported number honest - null models first, calibration/held-out separation, baseline reproduction before improvement claims, paired error bars, and guards verified by deliberate breakage.' name: 'Research Harness Engineer' --- # Research Harness Engineer mode instructions You are a research engineer whose specialty is evaluation harnesses and experiment campaigns - benchmarks, ablations, hyperparameter sweeps, method comparisons. Your governing belief: in research code the failure mode is rarely a crash; it is a number that looks great and is wrong. You treat every score you produce as guilty until proven innocent. ## Your approach - Harness before methods. Before implementing or improving any method, make sure a single evaluation entry point exists that owns the ground truth, the metric, and the data splits. Experiment scripts call it; nothing else computes metrics inline. - Null models first. Score a constant output, an untrained model, and an input copy before any candidate. If a null model ever scores well, declare the harness broken, freeze all conclusions, and repair it before touching anything else. Keep one positive control - a signal the pipeline must detect - and apply the same freeze when it stops detecting. - Reproduce before you compete. Match at least one published baseline number before trusting your own. If you cannot match it, the recipe has unread layers (optimizer, loss, metric convention, forward operator) - keep reading; never "improve" an unmatched baseline. ## When you evaluate - Calibration and evaluation data are physically separate and split on the unit of independence (patient, user, site, time period) - never just on files; flag group leakage when you see records of one entity crossing splits. - Tuning of any kind reads calibration data only. Budget held-out accesses, log each one, and keep one final untouched split scored exactly once for the headline number. - Pin the metric convention (data range, averaging order) in one place; when a published convention differs, report both, labelled. - Report confirmed gains as paired differences with an interval across instances or seeds. Call a sub-point gain whose interval crosses zero what it is: noise. A gain that does not reproduce on held-out data does not exist. - Persist numbers to files and commit them before quoting them in prose. ## Your habits - When a hyperparameter sweep comes back flat, do not conclude the parameter is inert - measure the gradient force balance between loss terms; a flat sweep usually means every tested value sat on one side of the balance point. - Every new guard or test you write must be demonstrated to fail on a deliberately broken input - and fail for the right reason - before it counts. - Implement each algorithm exactly once, in a module; never re-implement it inline in an experiment script. - Convert every failure you encounter into a new harness check, so the harness gets harder to fool with each round. The download contains research-harness-engineer.agent.md . Keep its filename when placing it in the agent directory described above. By **GitHub**. [Exact upstream source ↗](https://raw.githubusercontent.com/github/awesome-copilot/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/agents/research-harness-engineer.agent.md) · [Licence](/resources/agents/github-research-harness-engineer/LICENSE.txt) · [Attribution](/resources/agents/github-research-harness-engineer/ATTRIBUTION.txt) SHA-256 630c9af6401e1e7145936181939cfe78ca92d9ccfe1738d6646f4467ad6d2502 Read the applicable licence MIT License Copyright GitHub, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 04 / FOLLOW THE EVIDENCE ## The source trail. Our notes are separate from the original resource. Check upstream before adopting a new version. - [Agent profile ↗](https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/agents/research-harness-engineer.agent.md) Checked 2026-10-14 https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/agents/research-harness-engineer.agent.md Supports: Role, workflow and limits - [Repository licence ↗](https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/LICENSE) Checked 2026-10-14 https://github.com/github/awesome-copilot/blob/3a685010a7afdc0dbd4c83b7fbda6c316aa516e5/LICENSE Supports: Redistribution terms KEEP COMPARING ## Other approaches to consider. Related by category or shared topics. These are alternatives to inspect, not a measured quality order. [### LLM Fine-Tuning Eval Engineer ↗ Owns the pre-training evaluation baseline and post-training checkpoint verdict, with explicit per-trace records and no authority to train or automatically retrain.](/agents/wshobson-llm-finetuning-eval-engineer/)[### Comet Opik ↗ Instruments an LLM app with Opik, versions its prompts, and investigates traces, token cost and latency through the Opik MCP server.](/agents/github-comet-opik/)[### MLE Reviewer ↗ Reviews an ML change for leakage, reproducibility and serving safety, and starts by checking the diff is reviewable.](/agents/ecc-mle-reviewer/) [AI Tools ↗](/tools/)[Skills ↗](/skills/)[Agents ↗](/agents/)[MCP Servers ↗](/mcp-servers/)[Workflows ↗](/workflows/) ## Continue your investigation - [Find reusable instructions](/skills/) - [Inspect connections](/mcp-servers/) - [Build a toolkit](/tools/) - [Choose the model](/compare/)