SWE-Bench Pro
Real repository issues resolved end-to-end, graded by hidden tests. Four harnesses ran the identical task set on three models (deepseek-v4-flash, claude-sonnet-5, claude-opus-5).
Cheapest solve on every model · vs Claude Code, same model
$0.0498 / solve · 2.0× cheaper
Vetta posts the lowest cost per solved task on all three models. On deepseek-v4-flash it solves a task for $0.0498 — 2.0× cheaper than Claude Code on the same model ($0.1012) while resolving more of the suite (80.0% vs 66.7%).
Efficiency leaderboard?All harness × model cells — filter by model, click a column to re-sort.
Swapping the model changes the price of a token by two orders of magnitude; the harness decides how many of them a task needs. Only same-model rows are a like-for-like comparison.
Sorted by $ / task ↑
| # | Harness | Model | |||||
|---|---|---|---|---|---|---|---|
| 1 | Vetta | deepseek ◆ | $0.0398 | 80.0% | $0.0498 | 13 min | 21.9k |
| 2 | Pi | deepseek | $0.0482 | 80.0% | $0.0603 | 14 min | 24.5k |
| 3 | Hermes | deepseek ◆ | $0.0647 | 86.7% | $0.0746 | 18 min | 25.9k |
| 4 | Claude Code | deepseek | $0.0675 | 66.7% | $0.1012 | 11 min | 23.2k |
| 5 | Vetta | sonnet | $0.5538 | 66.7% | $0.8303 | 5 min | 14.3k |
| 6 | Pi | sonnet | $0.5865 | 60.0% | $0.9775 | 7 min | 13.4k |
| 7 | Vetta | opus | $0.8052 | 86.7% | $0.9287 | 6 min | 11.2k |
| 8 | Claude Code | sonnet ◆ | $1.0547 | 100.0% | $1.0547 | 8 min | 17.8k |
| 9 | Pi | opus | $1.2479 | 100.0% | $1.2479 | 9 min | 13.5k |
| 10 | Claude Code | opus | $1.7002 | 100.0% | $1.7002 | 10 min | 13.1k |
| 11 | Hermes | sonnet | $1.7736 | 66.7% | $2.6591 | 15 min | 32.2k |
| 12 | Hermes | opus | $3.6503 | 100.0% | $3.6503 | 22 min | 33.9k |
◆ on the cost-efficiency frontier — no other configuration is both cheaper and higher-scoring. Dollars are rate-card token costs for the listed model.
Tokens and turns?What a harness consumes per task: tokens read, tokens generated, model calls made.
What each harness actually consumes to finish a task: how much context it reads, how much it generates, and how many model calls it takes to get there.
Input tokens per task
Mean tokens read per attempt, cache reads included.
VettaOutput tokens per task
Mean tokens the model generates per attempt across the run.
VettaAgent turns per task
Mean model calls per attempt — how many steps a task takes.
VettaWhy we run this benchmark
Issue resolution is the workload most teams actually buy an agent for. Running four harnesses over the identical tasks on three models — from a budget open-weights model to the priciest frontier one — shows how much of the bill is the model and how much is the machinery around it, with Vetta benchmarked as just another row against the strongest public alternatives.
Methodology
Tasks come from SWE-Bench Pro, pinned to a fixed dataset commit. Each attempt starts from a clean containerized checkout of the repository at the issue commit and runs unattended; grading is the suite's hidden tests — binary and automatic, no partial credit, no human judging.
Every harness × model cell ran the identical task set at identical sampling settings. Dollars are rate-card token costs for the listed model, so same-model rows compare exactly; attempts that failed for infrastructure reasons are excluded rather than estimated.
Want the winning configuration? Deploy Vetta or explore the CLI.