---
title: "SWE-bench coding agents: measurement, limitations and evaluation guide · Undominated.ai"
canonical: https://undominated.ai/benchmarks/swe-bench/
description: "A coding result describes a model working through a particular agent and environment. Compare the full run configuration before treating a leaderboard entry as a model choice."
---

# SWE-bench coding agents: measurement, limitations and evaluation guide · Undominated.ai

> A coding result describes a model working through a particular agent and environment. Compare the full run configuration before treating a leaderboard entry as a model choice.

Evaluate repository issue resolution

# SWE-bench coding agents

A coding result describes a model working through a particular agent and environment. Compare the full run configuration before treating a leaderboard entry as a model choice.

Source reviewed 2026-09-23 · [Official SWE-bench leaderboards](https://www.swebench.com/)

 On this page [What it measures](#measurement)[Limitations](#limits)[Fair comparison](#controls)[Your evaluation](#experiment)

## What it measures

SWE-bench reports the fraction of task instances resolved. Its benchmark subsets differ. The official Bash Only view holds the mini-SWE-agent environment constant; other views can include different agents.

This is a method guide. Undominated does not publish or blend this benchmark’s scores into its model ranking.

## What the result does not establish

 - A result from one benchmark subset is not directly interchangeable with another subset’s percentage.
- Changing the agent, tools, retry policy or budget changes the system being evaluated.
- Public repository tasks do not establish success on your private codebase, security review or deployment workflow.

## Hold these conditions constant

 - Benchmark subset and evaluation revision
- Agent implementation, tool permissions and environment
- Model version, budget, retries and stopping rule
- Task-level outcomes and whether the run was independently checked

## An evaluation for your team

Use disposable repository fixtures with acceptance tests and a fixed tool budget. Record failed attempts and total spend, not just successful patches. Review correctness and maintainability before counting a task as accepted.

Write down the task version, attempt count, accepted outcomes, observed spend and latency. Preserve failures in the denominator. A new model version or tool configuration deserves a new record.

 [Inspect model references in a repository →](/check-repo/)[Attach your own evaluation results →](/compare/)[Review agent definitions →](/agents/)

Editorial guidance is not a new measurement. Follow the official source for its current benchmark definition and result history.

## Continue your investigation

 - [Inspect confidence intervals](/significance/)
- [Compare embedding evidence](/embeddings/)
- [Evaluate a shortlist](/compare/)
- [Read ranking rules](/methodology/)