Evidence
Two kinds of evidence, with different strengths. Neither is a benchmark leaderboard entry.
Read these before the numbers
- One model, mostly our own work. Every paired run used Gemini 3.8 Flash. 60 of the 66
tasks are in blyte's own two repositories (Rust and Python); 6 are from work repositories
that now use blyte. Plans were written by an orchestrating agent, often with the code
spelled out.
- FrontierHarness is one run on a different model. blyte ran on Gemini 3.8 Flash; the
leaderboard runs on Kimi K3. Local infrastructure, one attempt per task, 27 of 30 tasks
matched: about ±9 points of noise.
- "Mergeable" is a reviewer's judgement.
- Against a frontier model working alone, two tasks so far: see below.
- The paired runs measure blyte as it is meant to be used: Claude planned every task and
reviewed every result, and blyte implemented on a cheap model. They compare blyte with another
implementer, agy, in that same position.
- The FrontierHarness run measures blyte on its own: no plan, a cheap model and the raw
task, against harnesses that plan and implement for themselves. It is a hard test of the
implementer, not a picture of the intended use.
Paired shadow runs against agy
Method. Every change to blyte, and to the repository that evaluates it, was implemented twice
from the same commit, side by side on one machine: once by blyte and once by agy (Google's
Antigravity CLI), both on Gemini 3.8 Flash, both given the same plan and the same rules. Both
results were checked with the repository's full check command and reviewed; the better one was
merged. Every run, check and verdict is kept in a ledger. Only these paired runs count as
comparisons.
0.53×agy's time, medianfaster in 54 of 63
0.64×agy's cost, mediancheaper in 58 of 65 · list price
49/66blyte's result mergeable as-isagy: 51 of 66 · both passed the check in 62 of 66
0.63×agy's cost, mediancheaper in 35 of 37 · list price
30/37blyte's result mergeable as-isagy: 32 of 37 · both passed the check in 36 of 37
Time: blyte ÷ agy, per task
63 paired tasks · median 0.53× · below the line in 54 of 63
each bar is one task, sorted; hover for the taskscale 0 to 2× (taller is capped)
Cost: blyte ÷ agy, per task
65 paired tasks · median 0.64× · below the line in 58 of 65 · list price
each bar is one task, sorted; hover for the taskscale 0 to 2× (taller is capped)
Total wall time, all paired tasks
63 tasks, same model, same plans
Total model cost, all paired tasks
65 tasks · Gemini 3.8 Flash list price
What it shows. On real tasks (new providers, the sandbox, report fields, deleting thousands of
lines of dead code, CI), the same model under blyte produced work of the same quality as under agy
in about half the time, for about two thirds of the cost: 6.5 hours and $40.12 in total against
10.8 hours and $56.44. Three runs are left out of the time figures: a gate-slot bug (fixed the
next day) made blyte wait up to 27 minutes, or deadlocked both runs.
What it does not show.
- Most tasks are in two of our own repositories (Rust and Python), with 6 from work repositories,
and the plans were written by an orchestrating agent, often with the code spelled out.
Open-ended tasks may behave differently.
- "Mergeable as-is" is a reviewer's judgement, and it counts an equivalent result that was not
merged because the other was faster.
- Two runs were much slower for blyte (1.8× each). In one it wandered off to clear a host cache
after a gate failure; in the other a deleted session temp directory broke every later command.
Both led to fixes now in blyte: the workspace boundary and the write sandbox, and a
self-healing session. Other ratios above 1.5× are mostly runs where agy gave up early or made no
changes (up to 8× when agy's run failed after 53 s), so they flatter agy, not blyte.
- In four pairs only agy's result was unusable (no commit, or a crashed run); in two only blyte's:
the sandbox (above) and a flaky timing test that blyte loosened until it passed, without saying
so. In three neither was: the gate-slot bug, and two attempts at a 479-edit rename that a
reviewed script then did.
- One model. The comparison says nothing about blyte against agy on other models.
All 66 paired runs, task by task
| task |
agy |
blyte |
blyte ÷ agy (time, cost) |
result |
read returns whole files |
382 s, $0.47 |
266 s, $0.25 |
0.70×, 0.54× |
identical diffs |
| gate reports every failure |
460 s, $0.35 |
368 s, $0.24 |
0.80×, 0.70× |
identical diffs |
| read from the top returns the whole file |
320 s, $0.30 |
349 s, $0.16 |
1.09×, 0.55× |
identical diffs |
| quick checks before stopping |
138 s, $0.16 |
89 s, $0.11 |
0.64×, 0.66× |
identical diffs |
build stamp in --version and reports |
160 s, $0.21 |
89 s, $0.18 |
0.56×, 0.83× |
identical diffs |
| eval: remove the MCP arms (−2,680 lines) |
481 s, $0.98 |
288 s, $0.76 |
0.60×, 0.78× |
blyte's code better, wrong commit message |
| eval: remove the daemon and relay arms (−3,510) |
492 s, $0.71 |
231 s, $0.47 |
0.47×, 0.66× |
blyte better (agy dropped valid tests) |
| eval: remove the state-engine arms (−2,780) |
1030 s, $1.64 |
300 s, $0.84 |
0.29×, 0.51× |
blyte merged; agy stopped on a plan gap |
| run telemetry in the report |
394 s, $0.77 |
237 s, $0.42 |
0.60×, 0.55× |
same logic; agy's merged (doc comments) |
| eval: remove 34 dead launch scripts |
249 s, $0.36 |
116 s, $0.26 |
0.47×, 0.73× |
identical |
| OpenAI-compatible provider |
563 s, $0.73 |
256 s, $0.56 |
0.45×, 0.78× |
equal; blyte's merged |
read keeps long lines |
292 s, $0.42 |
113 s, $0.19 |
0.39×, 0.45× |
identical diffs |
| commit footer (an experiment, later removed) |
732 s, $1.10 |
266 s, $0.52 |
0.36×, 0.47× |
equal |
| Anthropic prompt caching |
372 s, $0.51 |
130 s, $0.24 |
0.35×, 0.46× |
near-identical |
| eval: time in the harness comparison |
1661 s, $0.74 |
3024 s, $0.47 |
1.82×, 0.63× |
agy's merged: blyte's run cleared a host cache after a gate failure |
| remove the commit footer |
216 s, $0.30 |
112 s, $0.19 |
0.52×, 0.63× |
blyte's clean revert merged |
| blyte states the workspace boundary |
300 s, $0.41 |
109 s, $0.16 |
0.36×, 0.40× |
identical |
| keep failed gate output in the report |
840 s, $1.36 |
365 s, $1.00 |
0.43×, 0.74× |
equal; both needed a plan fix |
| eval: CI workflow |
561 s, $0.46 |
125 s, $0.21 |
0.22×, 0.45× |
equal |
| eval: offline agent, online grader |
572 s, $0.98 |
303 s, $0.73 |
0.53×, 0.74× |
equal |
| honest completion prompt when ungated |
255 s, $0.34 |
118 s, $0.15 |
0.46×, 0.44× |
equal |
| ChatGPT-login provider |
714 s, $1.12 |
407 s, $0.80 |
0.57×, 0.72× |
equal |
| macOS write sandbox |
767 s, $1.51 |
1414 s, $4.30 |
1.84×, 2.85× |
agy's merged: blyte's session broke when its temp directory was deleted (a harness bug, fixed next) |
| session recreates its temp directory |
253 s, $0.34 |
291 s, $0.44 |
1.15×, 1.29× |
equal |
--max-cost |
514 s, $1.01 |
400 s, $0.98 |
0.78×, 0.97× |
agy's merged: blyte edited an existing test |
| sandbox refuses credential reads |
453 s, $0.72 |
186 s, $0.46 |
0.41×, 0.63× |
equal; both needed a compatibility fix |
| withhold credential-looking env vars |
549 s, $0.91 |
237 s, $0.54 |
0.43×, 0.60× |
equal |
| Linux Landlock sandbox |
632 s, $1.18 |
288 s, $0.71 |
0.46×, 0.60× |
equal |
| quota fails fast; login errors |
398 s, $0.60 |
207 s, $0.39 |
0.52×, 0.64× |
equal |
--network=off |
907 s, $1.51 |
293 s, $0.79 |
0.32×, 0.52× |
equal; blyte's commit message fixed |
| security settings in the report |
476 s, $0.81 |
177 s, $0.37 |
0.37×, 0.46× |
equal |
blyte dispatch and blyte review |
700 s, $1.14 |
349 s, $0.86 |
0.50×, 0.75× |
equal; blyte's merged |
blyte skill install |
307 s, $0.52 |
201 s, $0.29 |
0.65×, 0.56× |
equal; blyte's merged |
| login shell keeps the inherited PATH |
328 s, $0.48 |
177 s, $0.23 |
0.54×, 0.48× |
equal; blyte's merged |
skill --default, parallel dispatch |
481 s, $0.74 |
281 s, $0.44 |
0.58×, 0.60× |
equal; blyte's merged |
| skill default on, parallel first |
340 s, $0.48 |
210 s, $0.35 |
0.62×, 0.72× |
equal; blyte's merged |
| skill for Codex |
503 s, $0.83 |
245 s, $0.55 |
0.49×, 0.66× |
equal; blyte's merged |
blyte.toml and blyte init |
584 s, $1.05 |
272 s, $0.62 |
0.47×, 0.59× |
equal; blyte's merged |
| Linux read allowlist |
494 s, $0.83 |
305 s, $0.65 |
0.62×, 0.78× |
equal; blyte's merged |
| machine-wide gate slots |
564 s, $1.00 |
313 s, $0.74 |
0.55×, 0.74× |
equal; blyte's merged |
grep shows match-centred excerpts |
674 s, $1.07 |
368 s, $0.71 |
0.55×, 0.66× |
equal; blyte's merged (the plan's approach needed a follow-up) |
| failing gate steps first on a retry |
559 s, $0.75 |
232 s, $0.44 |
0.42×, 0.59× |
equal; blyte's merged |
blyte status |
615 s, $0.94 |
271 s, $0.60 |
0.44×, 0.64× |
equal; blyte's merged (agy's run ended in an error) |
| one sentence in the ungated text |
173 s, $0.24 |
124 s, $0.16 |
0.72×, 0.66× |
equal; blyte's merged |
eval: the dispatch_blyte tool |
456 s, $0.92 |
1872 s, $0.67 |
–, 0.73× |
equal; blyte's merged. Its time is not counted: 27 min waiting on a gate-slot bug, fixed next |
eval: the claude+blyte harness |
854 s, $1.35 |
1905 s, $0.84 |
–, 0.62× |
blyte's merged (more thorough). Its time is not counted: the same bug |
grep -E fallback, first try |
1923 s, $0.16 |
2084 s, – |
–, – |
neither: the gate-slot bug deadlocked both runs (not counted for time) |
| gate slots off in nested runs |
395 s, $0.49 |
207 s, $0.46 |
0.52×, 0.93× |
equal; blyte's merged |
grep -E fallback |
182 s, $0.21 |
291 s, $0.45 |
1.60×, 2.12× |
same fix; blyte's merged, and it found a real test bug |
| eval: cost columns |
242 s, $0.39 |
136 s, $0.18 |
0.56×, 0.46× |
equal; blyte's merged |
| eval: no subagents in the claude arms |
299 s, $0.43 |
187 s, $0.30 |
0.63×, 0.70× |
equal; blyte's merged |
| eval: plan-first mode |
491 s, $0.96 |
250 s, $0.50 |
0.51×, 0.52× |
blyte's merged after adding a log line agy had |
| a batching example in the prompt |
284 s, $0.36 |
146 s, $0.17 |
0.51×, 0.49× |
identical; blyte's merged |
| Vertex backend for Gemini |
3187 s, $0.73 |
425 s, $0.91 |
0.13×, 1.25× |
blyte's merged (one fix); agy produced nothing in 53 min |
read takes several paths |
216 s, $0.27 |
353 s, $0.58 |
1.63×, 2.17× |
blyte's merged; agy made no changes |
| eval: rename to blyte, first try |
2497 s, $4.63 |
939 s, $3.16 |
0.38×, 0.68× |
neither finished 479 hand edits |
| gate failures keep head and tail |
619 s, $1.00 |
462 s, $0.68 |
0.75×, 0.68× |
blyte's merged; agy made no commit |
| work repo: 128 workers, bigger head |
1443 s, $1.52 |
923 s, $0.98 |
0.64×, 0.64× |
same YAML; blyte's merged after removing an out-of-scope test edit |
| work repo: a per-build cost report |
1341 s, $1.80 |
1019 s, $0.90 |
0.76×, 0.50× |
both needed fixes; blyte's merged |
| eval: rename to blyte, second try |
2861 s, $5.26 |
1417 s, $2.37 |
0.50×, 0.45× |
neither finished; a reviewed script did it |
| eval: stripped guidance stays stripped |
144 s, $0.24 |
118 s, $0.18 |
0.82×, 0.75× |
identical; blyte's merged |
| work repo: straggler clock |
376 s, $0.40 |
540 s, $0.39 |
1.44×, 0.98× |
identical code; agy's merged (blyte added an out-of-scope test fixture) |
| eval: start-then-wait dispatch |
258 s, $0.31 |
427 s, $0.45 |
1.66×, 1.44× |
agy's merged: blyte loosened a flaky timing test and did not say so |
| a service: carry Redis read errors through |
53 s, $0.12 |
424 s, $0.73 |
8.00×, 6.31× |
blyte's merged; agy's run failed with a server error |
| work repo: modes and a per-run report |
807 s, $1.25 |
408 s, $0.54 |
0.51×, 0.43× |
equal; blyte's merged |
| work repo: seed modes |
760 s, $0.73 |
237 s, $0.15 |
0.31×, 0.20× |
blyte's merged |
In work repositories
Since 29 September blyte has also been the implementer in the author's work repositories, planned and
reviewed by Claude Code, with an A/B against agy on a sample of tasks (the six work-repository
pairs are in the table above).
- Reviewed runs (the dispatch ledger): 14 runs in three repositories, all reviewed: 8 merged
as-is, 6 needed a fix, none redone. Four of the six fixes were the same out-of-scope edit (an
AWS_PROFILE workaround in a test fixture, because the sandbox denies ~/.aws but passed the
profile variable); blyte now withholds that variable, and no run has made the edit since. The
other two: a missed try/except the plan asked for, and a wrong commit message.
- Unreviewed runs: 20 more dispatches in four repositories were started with
blyte dispatch
directly, outside the ledger, so they have no review verdict. By blyte's own reports, 18 passed
their gate and 2 ended in an error (a missing API key, before the key file); about $15.70 of
Gemini in total, a median of 7 minutes each. New sessions now route every task through the
ledger.
FrontierHarness tasks
FrontierHarness runs 12 harness configurations on Kimi K3 over 30
tasks (21 Terminal-Bench 2, 9 DeepSWE), one attempt each, graded by each task's held-out tests.
blyte ran the same tasks once, on Gemini 3.8 Flash, locally, offline, graded the same way.
On the 27 tasks that could be matched, blyte passed 19/27 (70%). The best published
configuration, Codex on Kimi K3, passed 17/27 (63%); the others passed 12 to 16. blyte solved two
tasks no configuration solved.
Read it as "competitive with the best harnesses", not "better". The model differs, the
infrastructure differs, each task ran once (about ±9 points of noise on 27 tasks), and three tasks
were excluded because their offline versions could not be reproduced. A same-model comparison
needs blyte on Kimi K3.
FrontierHarness: tasks passed, 27 matched tasks
bars scaled to all 27 tasks · one attempt each · blyte on a different model from the leaderboard's · see the caveats above
blyte · Gemini 3.8 Flash
19/27
Codex · Kimi K3
17/27
DSH Creator · Kimi K3
16/27
Claude Code · Kimi K3
16/27
Pi · Kimi K3
15/27
DSH PTC · Kimi K3
15/27
DSH Standard · Kimi K3
15/27
Oh My Pi · Kimi K3
14/27
Kimi Code · Kimi K3
14/27
DSH Minimal · Kimi K3
14/27
Exo · Kimi K3
13/27
Hermes · Kimi K3
13/27
OpenCode · Kimi K3
12/27
The full write-up, with per-task results and every caveat:
blyte on the FrontierHarness tasks.
What was measured along the way
Harness research records what was tried, what the
literature says, and what was removed because it did not pay for itself: a daemon, an MCP server,
a worktree engine, advisory review, a TDD mode, LLM-written fold notes, four losing context
strategies, and more.
Against Claude doing the work itself
The question. If Claude plans and reviews and blyte implements, how much Claude does that save,
and what does it cost in quality and time, compared with Claude doing the whole task?
Method. Two real-repository tasks from FrontierHarness, graded by held-out tests:
python-statemachine-state-data-scoping (Python) and katex-multicolumn-array-spans (TypeScript).
Claude Sonnet in Claude Code, two runs of each setup, interleaved. On the first task, three setups:
Claude implementing a written plan; Claude dispatching the same plan to blyte (Gemini 3.8 Flash)
and reviewing; and Claude on the raw task with no plan. One planning run wrote the plan both planned
setups used, so its cost falls on neither. Claude's own setup and plugins were kept out of every run.
What it showed.
- Planning decided the outcome. Both setups with the plan passed every run; Claude alone on the
raw task failed both, a few target tests short (one also broke three existing tests). Who implemented did not change the result.
- blyte cut Claude's usage by two thirds on the first task and by 86% on the second. With
blyte implementing, Claude spent a third as much, then about an eighth as much ($0.13 against
$0.90), as when it implemented the plan itself. Both differences were larger than the spread
between runs, and blyte's setup passed every run (Claude alone on the second task: one of two).
- The total cost depends on the task. On the first task, where Claude implementing cost about
$5.60, the hand-off was also about a quarter cheaper in total, counting blyte's Gemini cost. On
the second, Claude alone was quick and cheap (under $1 in 6–8 minutes), so the Gemini cost made
the hand-off about three times dearer in total. The Claude saving holds either way.
- blyte was slower, two and a half to three and a half times per task. Gemini works in many
small steps. In our own work, separate tasks dispatched at once finished about twice as fast
overall as they would have one after another (nearly three times with three or four at once);
Claude's part stays its plan and its review.
- Splitting one feature into parallel dispatches did not pay (two tasks, six runs with a split).
The parts were rarely independent and of similar size, and each dispatch re-reads the code it
touches, so splitting cost 30–160% more Gemini and was slower, even with the parts running at
once. One dispatch per feature was the fastest and cheapest. Parallelism pays across separate
tasks, not within one. (Claude Code runs one MCP server's tool calls one after another; the
evaluation's dispatch tool now starts dispatches in the background so they can overlap.)
Who writes the plan, and is the review needed? Three more setups on the same task, on Gemini
only, two runs each: blyte alone on the raw task; blyte writing its own plan and then implementing
it; and blyte implementing Claude's plan with no Claude review.
- A frontier plan with blyte implementing, unreviewed, was the best combination: it passed both
times and was the cheapest and fastest pass of any setup, about two dollars of Gemini and a
quarter of an hour, on top of the one-off plan.
- Claude's review added no quality here: with and without it, blyte's work passed; with it, the
task cost about twice as much and took three times as long. The gate did the checking.
- A weak plan is worse than none: from its own plan blyte failed both runs, a few target tests
short, while from the raw task it passed both.
- blyte alone beat Claude alone on this task, passing both runs at about two thirds of the cost.
On the FrontierHarness tasks blyte alone passed 5 of 9 of the same kind, so this varies by task.
So: spend the frontier model on the plan, let a passing gate do most of the checking, and keep
deep review for failed gates and risky changes. The pattern pays when frontier usage is the
constraint (a subscription's limits, or an API bill): it cut Claude's usage by two thirds to 86% on
both tasks. Run separate tasks in parallel; keep one feature in one dispatch. For one small task
that you want done quickly and cheaply in total, Claude alone is faster.
Write-ups are in the eval repository.
Known costs on Gemini. Gemini makes one tool call per turn, whatever the prompt asks, so blyte
on Gemini takes more turns than Claude does for the same work, and every turn re-sends the context.
Gemini's automatic prompt caching misses on roughly a quarter of blyte's turns. Both were
investigated; neither a prompt change, a multi-file read, streaming, explicit caching nor the Vertex
AI endpoint improved them, so none of those were kept.
Limits. One task and two runs per setup: a direction, not a rate. The plan was unusually
detailed, which favours delegation. Claude's cost is the notional list price that Claude Code reports under a subscription.
Results: blyte-eval results/offload-v2-sonnet-20260928-1028/ and results/offload-v3-blyte-20260928-1754/.