SWE-bench Deep Dive | Paper Reading | Bloss0m

← Back to Paper Reading

SWE-bench: Real GitHub Issues as Evaluation, but 1.96% Is Not a Model Ceiling

Intermediate Agent runtime, safety, and evaluation

SWE-bench Deep Dive: Part 1 , Note: Aug 27, 2026 , Paper: 2024 , AI Engineering

A source-grounded reading of Jimenez et al., ICLR 2024 Oral: the evaluation unit is a real GitHub issue, a full Python repository, and tests. Claude 2 resolves 1.96% under BM25; that number is a protocol, not a model ranking.

Understand it in 90 seconds

Read the full deep dive

For speaking invitations, internal engineering sessions, or architecture exchange, see the topics and public work I can bring into the conversation.

Speaking & contact