SWE-bench Deep Dive
Series · 1 posts
-
SWE-bench: Real GitHub Issues as Evaluation, but 1.96% Is Not a Model Ceiling
Intermediate Agent runtime, safety, and evaluationA source-grounded reading of Jimenez et al., ICLR 2024 Oral: the evaluation unit is a real GitHub issue, a full Python repository, and tests. Claude 2 resolves 1.96% under BM25; that number is a protocol, not a model ranking.
Understand it in 90 seconds
- Problem
- Coding benchmarks such as HumanEval compress success into writing a self-contained function. Real software engineering is: read a GitHub issue, edit a repository with thousands of files, and let tests decide whether the issue is fixed. Prior scores do not measure that job.
- Core insight
- Change the evaluation unit to a real issue plus a full Python repository plus tests. The model emits a patch; after unix patch applies it, every fail-to-pass and pass-to-pass test must pass before the instance is resolved. The changed control point is not a new agent architecture; it is what counts as success.
- Strongest evidence
- Under BM25 retrieval and a 13k context, Claude 2 resolves 1.96% (abstract, Section 1, Table 2). In the same protocol, Table 5 lists Claude 2 at 1.97% and also includes Claude 3 Opus at 3.79%. Under oracle retrieval, Claude 2 rises to 4.80% (Table 18). SWE-Llama reaches only 0.70% under BM25 and still mostly solves the simplest issues.
- Main boundary
- Python, issue-fix, binary tests. Resolve does not score maintainability, uncovered behavior, or review. BM25 and oracle are different retrieval conditions. Later SWE-bench Verified, SWE-agent, and ProMax use different setups, so their scores do not belong in this paper's tables.
For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.
Speaking & contact