Benchmarks

Zapier's business-workflow benchmark: 600 public tasks (Sales, Marketing, Operations, Support, Finance, HR; 100 each) on 47 simulated SaaS apps, graded by deterministic end-state assertions (strict pass rate). Official auto-bench harness, API toolset, 50 steps. Public set only: the zapier.com leaderboard uses a private, harder held-out set, so scores are not directly comparable.

100 end-to-end investment banking tasks (Excel models, pitch decks, memos) built with SEC EDGAR, market-data and logo tools, graded against ~110-criterion expert rubrics by the official agentic verifier (Gandalf, Gemini 3 Flash). Official Harbor tasks with OpenCode for every model, as on the v1.1 leaderboard; reports pass rate (all critical criteria met) and mean rubric score.

113 original long-horizon software engineering tasks across 91 open-source repos (Go, Python, TypeScript, Rust, JavaScript), graded by hidden behavioural tests. Official Pier runner on Modal; mini-swe-agent for every model, as on the leaderboard.

Harvey's Legal Agent Benchmark: 2,010 public long-horizon legal tasks (24 practice areas plus contracts, diligence and firm-knowledge) producing memos, redlines, spreadsheets and decks from a document set; every rubric criterion graded by two LLM judges (Claude Sonnet 4.6 + GPT-5.5), a task passes only if all criteria pass. Official LAB harness and evaluator. Public set only: the Vals leaderboard uses Harvey's private held-out set, so scores are not directly comparable.

Realistic health conversations graded against physician-written rubrics. Variants: full (5,000), hard (1,000) and consensus (3,671), reported separately. Official openai/simple-evals code with its GPT-5.4 (low reasoning) grader option.

2,500 expert-written closed-ended questions across dozens of subjects (multiple-choice and exact-match, ~14% with images), graded by an LLM judge. Official CAIS harness.

1,000 refined HLE questions, 500 reasoning and 500 knowledge, graded by an LLM judge. Official CAIS harness, no tools.

104 long-horizon tasks: generate a complete, runnable Python repository from a natural-language document, graded by the original project's tests (pass rate). Official OpenHands 0.56 harness inside a Docker-capable Modal VM per task.

Urban-planning map understanding (Chinese): 300 stratified questions over planning maps in 8 sub-tasks, each answer judged against expert critical points (score 0-2). Official zhuchichi56/PlanBench code; judge GPT-6 Luna (OpenAI's replacement for the retiring gpt-4o-mini).

200 tasks: rebuild a program from scratch given only its compiled binary and docs, graded by hidden behavioural tests. Official harness (mini-swe-agent + programbench eval) inside a Docker-capable Modal VM per task; mini-swe-agent for every model, as on the leaderboard. Headline: % resolved (all tests pass).

Contract-negotiation redlining: 140 tasks across 3 MSA negotiations and 4 turns, where the agent, as in-house counsel, produces a Word redline with tracked changes and comments. Graded against attorney-authored weighted rubrics by a 3-judge LLM panel (GPT-5.4 mini, Claude Haiku 4.5, Gemini 3.1 Flash-Lite; majority vote). Official Harbor tasks on Modal; Claude Code for Anthropic models, Codex for OpenAI models, Terminus-2 otherwise.

70 real-world refactoring tasks across 10 open-source repositories. A trial passes when the repository's tests still pass and an LLM judge confirms every must-have rubric. Official Harbor runner on Modal; Claude Code for Anthropic models, Codex for OpenAI models, mini-swe-agent otherwise, as on Scale's leaderboard.

20 long-horizon whole-repository stack migrations (language, framework, platform, build toolchain), graded by a three-stage ladder: model audit gate, behavioural suite (40), six adversarial verifier models (60). Official Harbor harness in Docker-capable Modal VMs; Claude Code or Codex per the benchmark's pins.

170 large multi-file refactoring issues (created after 2025) in C, C++, Go, Java, Python, Rust and TypeScript repos; official evaluator, mini-swe-agent with the paper's 300-step / $10 limits.

Real GitHub issues with screenshots and diagrams from JavaScript/TypeScript repos, graded by the repos' tests (% resolved). SWE-agent's multimodal config for every model; images are shown to the model.

100 real repositories in 22 languages, each at a commit holding many real bugs (4,068 in total). With no hints, the agent has 12 hours per repository to find and fix as many as it can; hidden per-bug tests decide, and any regression zeroes the repository. Official Harbor tasks on Modal with mini-swe-agent for every model, as on swesweep.com.

30 long-horizon enterprise SaaS engineering tasks (8 languages, 6 databases): build a production backend from a PRD, verified by a dependency-aware DAG evaluator (HTTP, SQL, browser, LLM judge). Official runners inside a Docker-capable Modal VM per task.

66 hard, realistic terminal tasks (software engineering, science, ML, operations, hardware, security, media), each checked by a hidden verifier. Official Harbor runner on Modal; Claude Code for Anthropic models, Codex for OpenAI models, Terminus-2 otherwise, as on the tbench.ai leaderboard.

70 expert-curated scientific research workflows across the life, physical, earth, mathematical and engineering sciences, each checked by an offline verifier. Official Harbor runner on Modal; Claude Code for Anthropic models, Codex for OpenAI models, Terminus-2 otherwise, as on the Terminal-Bench-Science leaderboard.