Skip to content

Evidence#

Two kinds of evidence, with different strengths. Neither is a benchmark leaderboard entry.

Read these before the numbers

  • One model, mostly our own work. Every paired run used Gemini 3.8 Flash. 60 of the 66 tasks are in blyte's own two repositories (Rust and Python); 6 are from work repositories that now use blyte. Plans were written by an orchestrating agent, often with the code spelled out.
  • FrontierHarness is one run on a different model. blyte ran on Gemini 3.8 Flash; the leaderboard runs on Kimi K3. Local infrastructure, one attempt per task, 27 of 30 tasks matched: about ±9 points of noise.
  • "Mergeable" is a reviewer's judgement.
  • Against a frontier model working alone, two tasks so far: see below.
  • The paired runs measure blyte as it is meant to be used: Claude planned every task and reviewed every result, and blyte implemented on a cheap model. They compare blyte with another implementer, agy, in that same position.
  • The FrontierHarness run measures blyte on its own: no plan, a cheap model and the raw task, against harnesses that plan and implement for themselves. It is a hard test of the implementer, not a picture of the intended use.

Paired shadow runs against agy#

Method. Every change to blyte, and to the repository that evaluates it, was implemented twice from the same commit, side by side on one machine: once by blyte and once by agy (Google's Antigravity CLI), both on Gemini 3.8 Flash, both given the same plan and the same rules. Both results were checked with the repository's full check command and reviewed; the better one was merged. Every run, check and verdict is kept in a ledger. Only these paired runs count as comparisons.

0.53×agy's time, medianfaster in 54 of 63
0.64×agy's cost, mediancheaper in 58 of 65 · list price
49/66blyte's result mergeable as-isagy: 51 of 66 · both passed the check in 62 of 66
0.63×agy's cost, mediancheaper in 35 of 37 · list price
30/37blyte's result mergeable as-isagy: 32 of 37 · both passed the check in 36 of 37

Time: blyte ÷ agy, per task

63 paired tasks · median 0.53× · below the line in 54 of 63
1.0 = same as agy
each bar is one task, sorted; hover for the taskscale 0 to 2× (taller is capped)

Cost: blyte ÷ agy, per task

65 paired tasks · median 0.64× · below the line in 58 of 65 · list price
1.0 = same as agy
each bar is one task, sorted; hover for the taskscale 0 to 2× (taller is capped)

Total wall time, all paired tasks

63 tasks, same model, same plans
agy
10.8 h
blyte
6.5 h

Total model cost, all paired tasks

65 tasks · Gemini 3.8 Flash list price
agy
$56.44
blyte
$40.12

What it shows. On real tasks (new providers, the sandbox, report fields, deleting thousands of lines of dead code, CI), the same model under blyte produced work of the same quality as under agy in about half the time, for about two thirds of the cost: 6.5 hours and $40.12 in total against 10.8 hours and $56.44. Three runs are left out of the time figures: a gate-slot bug (fixed the next day) made blyte wait up to 27 minutes, or deadlocked both runs.

What it does not show.

  • Most tasks are in two of our own repositories (Rust and Python), with 6 from work repositories, and the plans were written by an orchestrating agent, often with the code spelled out. Open-ended tasks may behave differently.
  • "Mergeable as-is" is a reviewer's judgement, and it counts an equivalent result that was not merged because the other was faster.
  • Two runs were much slower for blyte (1.8× each). In one it wandered off to clear a host cache after a gate failure; in the other a deleted session temp directory broke every later command. Both led to fixes now in blyte: the workspace boundary and the write sandbox, and a self-healing session. Other ratios above 1.5× are mostly runs where agy gave up early or made no changes (up to 8× when agy's run failed after 53 s), so they flatter agy, not blyte.
  • In four pairs only agy's result was unusable (no commit, or a crashed run); in two only blyte's: the sandbox (above) and a flaky timing test that blyte loosened until it passed, without saying so. In three neither was: the gate-slot bug, and two attempts at a 479-edit rename that a reviewed script then did.
  • One model. The comparison says nothing about blyte against agy on other models.
All 66 paired runs, task by task
task agy blyte blyte ÷ agy (time, cost) result
read returns whole files 382 s, $0.47 266 s, $0.25 0.70×, 0.54× identical diffs
gate reports every failure 460 s, $0.35 368 s, $0.24 0.80×, 0.70× identical diffs
read from the top returns the whole file 320 s, $0.30 349 s, $0.16 1.09×, 0.55× identical diffs
quick checks before stopping 138 s, $0.16 89 s, $0.11 0.64×, 0.66× identical diffs
build stamp in --version and reports 160 s, $0.21 89 s, $0.18 0.56×, 0.83× identical diffs
eval: remove the MCP arms (−2,680 lines) 481 s, $0.98 288 s, $0.76 0.60×, 0.78× blyte's code better, wrong commit message
eval: remove the daemon and relay arms (−3,510) 492 s, $0.71 231 s, $0.47 0.47×, 0.66× blyte better (agy dropped valid tests)
eval: remove the state-engine arms (−2,780) 1030 s, $1.64 300 s, $0.84 0.29×, 0.51× blyte merged; agy stopped on a plan gap
run telemetry in the report 394 s, $0.77 237 s, $0.42 0.60×, 0.55× same logic; agy's merged (doc comments)
eval: remove 34 dead launch scripts 249 s, $0.36 116 s, $0.26 0.47×, 0.73× identical
OpenAI-compatible provider 563 s, $0.73 256 s, $0.56 0.45×, 0.78× equal; blyte's merged
read keeps long lines 292 s, $0.42 113 s, $0.19 0.39×, 0.45× identical diffs
commit footer (an experiment, later removed) 732 s, $1.10 266 s, $0.52 0.36×, 0.47× equal
Anthropic prompt caching 372 s, $0.51 130 s, $0.24 0.35×, 0.46× near-identical
eval: time in the harness comparison 1661 s, $0.74 3024 s, $0.47 1.82×, 0.63× agy's merged: blyte's run cleared a host cache after a gate failure
remove the commit footer 216 s, $0.30 112 s, $0.19 0.52×, 0.63× blyte's clean revert merged
blyte states the workspace boundary 300 s, $0.41 109 s, $0.16 0.36×, 0.40× identical
keep failed gate output in the report 840 s, $1.36 365 s, $1.00 0.43×, 0.74× equal; both needed a plan fix
eval: CI workflow 561 s, $0.46 125 s, $0.21 0.22×, 0.45× equal
eval: offline agent, online grader 572 s, $0.98 303 s, $0.73 0.53×, 0.74× equal
honest completion prompt when ungated 255 s, $0.34 118 s, $0.15 0.46×, 0.44× equal
ChatGPT-login provider 714 s, $1.12 407 s, $0.80 0.57×, 0.72× equal
macOS write sandbox 767 s, $1.51 1414 s, $4.30 1.84×, 2.85× agy's merged: blyte's session broke when its temp directory was deleted (a harness bug, fixed next)
session recreates its temp directory 253 s, $0.34 291 s, $0.44 1.15×, 1.29× equal
--max-cost 514 s, $1.01 400 s, $0.98 0.78×, 0.97× agy's merged: blyte edited an existing test
sandbox refuses credential reads 453 s, $0.72 186 s, $0.46 0.41×, 0.63× equal; both needed a compatibility fix
withhold credential-looking env vars 549 s, $0.91 237 s, $0.54 0.43×, 0.60× equal
Linux Landlock sandbox 632 s, $1.18 288 s, $0.71 0.46×, 0.60× equal
quota fails fast; login errors 398 s, $0.60 207 s, $0.39 0.52×, 0.64× equal
--network=off 907 s, $1.51 293 s, $0.79 0.32×, 0.52× equal; blyte's commit message fixed
security settings in the report 476 s, $0.81 177 s, $0.37 0.37×, 0.46× equal
blyte dispatch and blyte review 700 s, $1.14 349 s, $0.86 0.50×, 0.75× equal; blyte's merged
blyte skill install 307 s, $0.52 201 s, $0.29 0.65×, 0.56× equal; blyte's merged
login shell keeps the inherited PATH 328 s, $0.48 177 s, $0.23 0.54×, 0.48× equal; blyte's merged
skill --default, parallel dispatch 481 s, $0.74 281 s, $0.44 0.58×, 0.60× equal; blyte's merged
skill default on, parallel first 340 s, $0.48 210 s, $0.35 0.62×, 0.72× equal; blyte's merged
skill for Codex 503 s, $0.83 245 s, $0.55 0.49×, 0.66× equal; blyte's merged
blyte.toml and blyte init 584 s, $1.05 272 s, $0.62 0.47×, 0.59× equal; blyte's merged
Linux read allowlist 494 s, $0.83 305 s, $0.65 0.62×, 0.78× equal; blyte's merged
machine-wide gate slots 564 s, $1.00 313 s, $0.74 0.55×, 0.74× equal; blyte's merged
grep shows match-centred excerpts 674 s, $1.07 368 s, $0.71 0.55×, 0.66× equal; blyte's merged (the plan's approach needed a follow-up)
failing gate steps first on a retry 559 s, $0.75 232 s, $0.44 0.42×, 0.59× equal; blyte's merged
blyte status 615 s, $0.94 271 s, $0.60 0.44×, 0.64× equal; blyte's merged (agy's run ended in an error)
one sentence in the ungated text 173 s, $0.24 124 s, $0.16 0.72×, 0.66× equal; blyte's merged
eval: the dispatch_blyte tool 456 s, $0.92 1872 s, $0.67 –, 0.73× equal; blyte's merged. Its time is not counted: 27 min waiting on a gate-slot bug, fixed next
eval: the claude+blyte harness 854 s, $1.35 1905 s, $0.84 –, 0.62× blyte's merged (more thorough). Its time is not counted: the same bug
grep -E fallback, first try 1923 s, $0.16 2084 s, – –, – neither: the gate-slot bug deadlocked both runs (not counted for time)
gate slots off in nested runs 395 s, $0.49 207 s, $0.46 0.52×, 0.93× equal; blyte's merged
grep -E fallback 182 s, $0.21 291 s, $0.45 1.60×, 2.12× same fix; blyte's merged, and it found a real test bug
eval: cost columns 242 s, $0.39 136 s, $0.18 0.56×, 0.46× equal; blyte's merged
eval: no subagents in the claude arms 299 s, $0.43 187 s, $0.30 0.63×, 0.70× equal; blyte's merged
eval: plan-first mode 491 s, $0.96 250 s, $0.50 0.51×, 0.52× blyte's merged after adding a log line agy had
a batching example in the prompt 284 s, $0.36 146 s, $0.17 0.51×, 0.49× identical; blyte's merged
Vertex backend for Gemini 3187 s, $0.73 425 s, $0.91 0.13×, 1.25× blyte's merged (one fix); agy produced nothing in 53 min
read takes several paths 216 s, $0.27 353 s, $0.58 1.63×, 2.17× blyte's merged; agy made no changes
eval: rename to blyte, first try 2497 s, $4.63 939 s, $3.16 0.38×, 0.68× neither finished 479 hand edits
gate failures keep head and tail 619 s, $1.00 462 s, $0.68 0.75×, 0.68× blyte's merged; agy made no commit
work repo: 128 workers, bigger head 1443 s, $1.52 923 s, $0.98 0.64×, 0.64× same YAML; blyte's merged after removing an out-of-scope test edit
work repo: a per-build cost report 1341 s, $1.80 1019 s, $0.90 0.76×, 0.50× both needed fixes; blyte's merged
eval: rename to blyte, second try 2861 s, $5.26 1417 s, $2.37 0.50×, 0.45× neither finished; a reviewed script did it
eval: stripped guidance stays stripped 144 s, $0.24 118 s, $0.18 0.82×, 0.75× identical; blyte's merged
work repo: straggler clock 376 s, $0.40 540 s, $0.39 1.44×, 0.98× identical code; agy's merged (blyte added an out-of-scope test fixture)
eval: start-then-wait dispatch 258 s, $0.31 427 s, $0.45 1.66×, 1.44× agy's merged: blyte loosened a flaky timing test and did not say so
a service: carry Redis read errors through 53 s, $0.12 424 s, $0.73 8.00×, 6.31× blyte's merged; agy's run failed with a server error
work repo: modes and a per-run report 807 s, $1.25 408 s, $0.54 0.51×, 0.43× equal; blyte's merged
work repo: seed modes 760 s, $0.73 237 s, $0.15 0.31×, 0.20× blyte's merged

In work repositories#

Since 29 September blyte has also been the implementer in the author's work repositories, planned and reviewed by Claude Code, with an A/B against agy on a sample of tasks (the six work-repository pairs are in the table above).

  • Reviewed runs (the dispatch ledger): 14 runs in three repositories, all reviewed: 8 merged as-is, 6 needed a fix, none redone. Four of the six fixes were the same out-of-scope edit (an AWS_PROFILE workaround in a test fixture, because the sandbox denies ~/.aws but passed the profile variable); blyte now withholds that variable, and no run has made the edit since. The other two: a missed try/except the plan asked for, and a wrong commit message.
  • Unreviewed runs: 20 more dispatches in four repositories were started with blyte dispatch directly, outside the ledger, so they have no review verdict. By blyte's own reports, 18 passed their gate and 2 ended in an error (a missing API key, before the key file); about $15.70 of Gemini in total, a median of 7 minutes each. New sessions now route every task through the ledger.

FrontierHarness tasks#

FrontierHarness runs 12 harness configurations on Kimi K3 over 30 tasks (21 Terminal-Bench 2, 9 DeepSWE), one attempt each, graded by each task's held-out tests. blyte ran the same tasks once, on Gemini 3.8 Flash, locally, offline, graded the same way.

On the 27 tasks that could be matched, blyte passed 19/27 (70%). The best published configuration, Codex on Kimi K3, passed 17/27 (63%); the others passed 12 to 16. blyte solved two tasks no configuration solved.

Read it as "competitive with the best harnesses", not "better". The model differs, the infrastructure differs, each task ran once (about ±9 points of noise on 27 tasks), and three tasks were excluded because their offline versions could not be reproduced. A same-model comparison needs blyte on Kimi K3.

FrontierHarness: tasks passed, 27 matched tasks

bars scaled to all 27 tasks · one attempt each · blyte on a different model from the leaderboard's · see the caveats above
blyte · Gemini 3.8 Flash
19/27
Codex · Kimi K3
17/27
DSH Creator · Kimi K3
16/27
Claude Code · Kimi K3
16/27
Pi · Kimi K3
15/27
DSH PTC · Kimi K3
15/27
DSH Standard · Kimi K3
15/27
Oh My Pi · Kimi K3
14/27
Kimi Code · Kimi K3
14/27
DSH Minimal · Kimi K3
14/27
Exo · Kimi K3
13/27
Hermes · Kimi K3
13/27
OpenCode · Kimi K3
12/27

The full write-up, with per-task results and every caveat: blyte on the FrontierHarness tasks.

What was measured along the way#

Harness research records what was tried, what the literature says, and what was removed because it did not pay for itself: a daemon, an MCP server, a worktree engine, advisory review, a TDD mode, LLM-written fold notes, four losing context strategies, and more.

Against Claude doing the work itself#

The question. If Claude plans and reviews and blyte implements, how much Claude does that save, and what does it cost in quality and time, compared with Claude doing the whole task?

Method. Two real-repository tasks from FrontierHarness, graded by held-out tests: python-statemachine-state-data-scoping (Python) and katex-multicolumn-array-spans (TypeScript). Claude Sonnet in Claude Code, two runs of each setup, interleaved. On the first task, three setups: Claude implementing a written plan; Claude dispatching the same plan to blyte (Gemini 3.8 Flash) and reviewing; and Claude on the raw task with no plan. One planning run wrote the plan both planned setups used, so its cost falls on neither. Claude's own setup and plugins were kept out of every run.

What it showed.

  • Planning decided the outcome. Both setups with the plan passed every run; Claude alone on the raw task failed both, a few target tests short (one also broke three existing tests). Who implemented did not change the result.
  • blyte cut Claude's usage by two thirds on the first task and by 86% on the second. With blyte implementing, Claude spent a third as much, then about an eighth as much ($0.13 against $0.90), as when it implemented the plan itself. Both differences were larger than the spread between runs, and blyte's setup passed every run (Claude alone on the second task: one of two).
  • The total cost depends on the task. On the first task, where Claude implementing cost about $5.60, the hand-off was also about a quarter cheaper in total, counting blyte's Gemini cost. On the second, Claude alone was quick and cheap (under $1 in 6–8 minutes), so the Gemini cost made the hand-off about three times dearer in total. The Claude saving holds either way.
  • blyte was slower, two and a half to three and a half times per task. Gemini works in many small steps. In our own work, separate tasks dispatched at once finished about twice as fast overall as they would have one after another (nearly three times with three or four at once); Claude's part stays its plan and its review.
  • Splitting one feature into parallel dispatches did not pay (two tasks, six runs with a split). The parts were rarely independent and of similar size, and each dispatch re-reads the code it touches, so splitting cost 30–160% more Gemini and was slower, even with the parts running at once. One dispatch per feature was the fastest and cheapest. Parallelism pays across separate tasks, not within one. (Claude Code runs one MCP server's tool calls one after another; the evaluation's dispatch tool now starts dispatches in the background so they can overlap.)

Who writes the plan, and is the review needed? Three more setups on the same task, on Gemini only, two runs each: blyte alone on the raw task; blyte writing its own plan and then implementing it; and blyte implementing Claude's plan with no Claude review.

  • A frontier plan with blyte implementing, unreviewed, was the best combination: it passed both times and was the cheapest and fastest pass of any setup, about two dollars of Gemini and a quarter of an hour, on top of the one-off plan.
  • Claude's review added no quality here: with and without it, blyte's work passed; with it, the task cost about twice as much and took three times as long. The gate did the checking.
  • A weak plan is worse than none: from its own plan blyte failed both runs, a few target tests short, while from the raw task it passed both.
  • blyte alone beat Claude alone on this task, passing both runs at about two thirds of the cost. On the FrontierHarness tasks blyte alone passed 5 of 9 of the same kind, so this varies by task.

So: spend the frontier model on the plan, let a passing gate do most of the checking, and keep deep review for failed gates and risky changes. The pattern pays when frontier usage is the constraint (a subscription's limits, or an API bill): it cut Claude's usage by two thirds to 86% on both tasks. Run separate tasks in parallel; keep one feature in one dispatch. For one small task that you want done quickly and cheaply in total, Claude alone is faster.

Write-ups are in the eval repository.

Known costs on Gemini. Gemini makes one tool call per turn, whatever the prompt asks, so blyte on Gemini takes more turns than Claude does for the same work, and every turn re-sends the context. Gemini's automatic prompt caching misses on roughly a quarter of blyte's turns. Both were investigated; neither a prompt change, a multi-file read, streaming, explicit caching nor the Vertex AI endpoint improved them, so none of those were kept.

Limits. One task and two runs per setup: a direction, not a rate. The plan was unusually detailed, which favours delegation. Claude's cost is the notional list price that Claude Code reports under a subscription. Results: blyte-eval results/offload-v2-sonnet-20260928-1028/ and results/offload-v3-blyte-20260928-1754/.