SWE-bench coding agents

What it measures

SWE-bench reports the fraction of task instances resolved. Its benchmark subsets differ. The official Bash Only view holds the mini-SWE-agent environment constant; other views can include different agents.

This is a method guide. Undominated does not publish or blend this benchmark’s scores into its model ranking.

What the result does not establish

• A result from one benchmark subset is not directly interchangeable with another subset’s percentage.
• Changing the agent, tools, retry policy or budget changes the system being evaluated.
• Public repository tasks do not establish success on your private codebase, security review or deployment workflow.

Hold these conditions constant

• Benchmark subset and evaluation revision
• Agent implementation, tool permissions and environment
• Model version, budget, retries and stopping rule
• Task-level outcomes and whether the run was independently checked

An evaluation for your team

Use disposable repository fixtures with acceptance tests and a fixed tool budget. Record failed attempts and total spend, not just successful patches. Review correctness and maintainability before counting a task as accepted.

Write down the task version, attempt count, accepted outcomes, observed spend and latency. Preserve failures in the denominator. A new model version or tool configuration deserves a new record.