Benchmarks

113 original long-horizon software engineering tasks across 91 open-source repos (Go, Python, TypeScript, Rust, JavaScript), graded by hidden behavioural tests. Official Pier runner on Modal; mini-swe-agent for every model, as on the leaderboard.

104 long-horizon tasks: generate a complete, runnable Python repository from a natural-language document, graded by the original project's tests (pass rate). Official OpenHands 0.56 harness inside a Docker-capable Modal VM per task.

200 tasks: rebuild a program from scratch given only its compiled binary and docs, graded by hidden behavioural tests. Official harness (mini-swe-agent + programbench eval) inside a Docker-capable Modal VM per task; mini-swe-agent for every model, as on the leaderboard. Headline: % resolved (all tests pass).

70 real-world refactoring tasks across 10 open-source repositories. A trial passes when the repository's tests still pass and an LLM judge confirms every must-have rubric. Official Harbor runner on Modal; Claude Code for Anthropic models, Codex for OpenAI models, mini-swe-agent otherwise, as on Scale's leaderboard.

20 long-horizon whole-repository stack migrations (language, framework, platform, build toolchain), graded by a three-stage ladder: model audit gate, behavioural suite (40), six adversarial verifier models (60). Official Harbor harness in Docker-capable Modal VMs; Claude Code or Codex per the benchmark's pins.

170 large multi-file refactoring issues (created after 2025) in C, C++, Go, Java, Python, Rust and TypeScript repos; official evaluator, mini-swe-agent with the paper's 300-step / $10 limits.

Real GitHub issues with screenshots and diagrams from JavaScript/TypeScript repos, graded by the repos' tests (% resolved). SWE-agent's multimodal config for every model; images are shown to the model.

100 real repositories in 22 languages, each at a commit holding many real bugs (4,068 in total). With no hints, the agent has 12 hours per repository to find and fix as many as it can; hidden per-bug tests decide, and any regression zeroes the repository. Official Harbor tasks on Modal with mini-swe-agent for every model, as on swesweep.com.

30 long-horizon enterprise SaaS engineering tasks (8 languages, 6 databases): build a production backend from a PRD, verified by a dependency-aware DAG evaluator (HTTP, SQL, browser, LLM judge). Official runners inside a Docker-capable Modal VM per task.

66 hard, realistic terminal tasks (software engineering, science, ML, operations, hardware, security, media), each checked by a hidden verifier. Official Harbor runner on Modal; Claude Code for Anthropic models, Codex for OpenAI models, Terminus-2 otherwise, as on the tbench.ai leaderboard.