PRDBench
Product requirements turned into working software. Every published harness ran the same five tasks with GLM-5.3-Flash, then one pinned judge scored every implementation criterion.
Cost per successful task · same model and task slice
$0.0194 / success · 7% less than Codex
Vetta cleared the success threshold on four of five tasks versus three each for Codex, Hermes, and Claude Code. This table only shows harnesses it beat on both cost and mean score, so it is not a full ranking.
Efficiency leaderboard?Published harness results on the same benchmark configuration, ordered by measured model cost per successful task.
Agent-model cost per successful task. The field shows only the harnesses Vetta beat on both this measure and mean rubric score. Every row used the same provider rate card.
Vetta$0.0194| Harness | Cost / successful task | Tasks solved | Success rate | Time / task | Input tokens / task | Output tokens / task | Turns / task |
|---|---|---|---|---|---|---|---|
| Vetta | $0.0194 | 4 / 5 | 80.0% | 11.0 min | 618.0k | 17.3k | 30.8 |
| Codex | $0.0208 | 3 / 5 | 60.0% | 6.3 min | 543.8k | 9.5k | — |
| Hermes | $0.0600 | 3 / 5 | 60.0% | 12.2 min | 1.8M | 20.2k | 40.2 |
| Claude Code | $0.128 | 3 / 5 | 60.0% | 34.4 min | 3.4M | 72.2k | 56.6 |
Success means a task score of at least 0.25. Mean score is the mean normalized rubric score, not the binary success rate. This is a five-task study, not the full PRDBench leaderboard. Figures come from our internal benchmark database. The trial-level rows are not public yet.
Why we run this benchmark
A product agent has to translate a written specification into an implementation that survives detailed evaluation, not merely produce a plausible patch. PRDBench measures that complete loop and exposes the cost of the harness around a fixed model.
Methodology
We ran task IDs 41, 32, 11, 24, and 16 from a content-pinned PRDBench slice. Each attempt began in its own workspace with a 45-minute limit. One DeepSeek V3.2 judge configuration regraded all 25 collected trials; the agents were not rerun during regrading.
The score for a task is the mean of its normalized evaluation criteria; success is a score of at least 0.25. Cost comes from measured uncached input, cache-read input, and output tokens at the fixed GLM-5.3-Flash rate card.
We ran more than one Vetta build and show the strongest, our E05 build. It is on a branch and not yet in the shipped product, and its held-out check had not run. The table leaves out any harness it did not beat on both mean score and cost per successful task.
Want to try it? Deploy Vetta or explore the CLI.