AgentSwarm Blog · Part 3 of 12

Four Agent CLIs, One Task: The Benchmark Nobody Runs

Part 3 of the AgentSwarm series — substrate economics.


The problem with agent-CLI benchmarks

Most "which agent CLI is best" comparisons measure vibes: how the output reads, whether the demo worked. Almost none answer the question that decides whether you can run a swarm at all: cost per successful turn, on an identical task, measured from a forensic record.

Because the arena records everything per turn — tool calls, thinking tokens, wall clock, dollars — the controlled comparison fell out of the system almost for free. Same task, same budget, four harnesses. Later I formalized it as benchmark.mjs: one fixed question ("Is commercial fusion power achievable by 2040?"), one turn per run, three repeats per substrate.

The fingerprint table

From Phase 1 (same swarm task, four substrates running as peers):

SubstrateModelSuccessReported $/turnTools/turnThink tok/turn
agygemini-3.8-flash-high100%$0.002213,368
ompgemini-3.8-flash100%$0.03097135,718
stepstep-5-preview20%$0.00114—
pideepseek-v4p1-flash50%$0.000653—

And the controlled benchmark (fusion question, 3 repeats each):

SubstrateRunsSuccess$/turnTools/turnThink tok/turn
agy3/3100%$0.0025.721,296
omp3/3100%$0.022872.038,472

Two immediately interesting things: omp issues ~3× the tool calls of agy for the same task, and pi costs ~66× less per turn than omp. But the naive reading of this table is wrong in an instructive way.

"$0.00" is doing a lot of work

Reported cost alone is misleading, because three of the four substrates report $0.00 for reasons that are not "free". The ledger records token counts, so the real economics are visible:

SubstrateReportedWhyFresh input tokensCache readCache hit
agy$0.00Google AI subscription2,244,4886,083,75673.0%
step$0.00Preview promo period3,008139,52097.9%
omp$0.1854pay-per-token99,228379,42579.3%
pi$0.0012Fireworks85036,74097.7%

Across Phase 1: total reported spend $0.1865; notional list-rate value of the same tokens $1.97; absorbed by subscriptions and promos $1.79.

agy did ~22× more fresh-token volume than omp and cost nothing. The correct conclusion is not "avoid omp" — it's that agy is dramatically underpriced on a flat subscription and should be the swarm's primary worker, not one peer among four. That decision directly shaped every later phase.

And notice the cache-hit column: step and pi re-send a near-identical context on every one of their many tool round-trips, which is why their hit rates approach 98%. Cached reads are charged, but ~30× cheaper than fresh input — that's why 26.2M session tokens cost a tenth of a cent instead of several dollars. If you're budgeting a swarm, cache-hit rate is a first-class architectural variable, not an implementation detail.

Neither failure was a capability failure

pi failed every turn in the first run. step succeeded once in five attempts in the second. Both look damning in the table. Both are scheduling problems, not model problems:

difference for comparable work — because it streams one event per token. Combined with 100–175 tool calls per turn, one step turn generates 6–10 MB of events and runs 15–25 minutes into a 7-minute timeout.

Both substrates are exhaustive to the point of self-truncation. They attempt far more work per turn than the wall clock allows, and die mid-sentence.

Four fixes, all verified against the ledger:

  1. Event pruning — keep structural events, drop per-token deltas
  2. Per-substrate timeouts — 25 min for step vs 7 min default
  3. Truncation preservation — a cut-off turn keeps its artifacts and logs a

warn instead of discarding the work (this single fix saved four files)

  1. An efficiency directive in every prompt, plus failure-aware scheduling

that deprioritizes repeatedly-failing agents

Result: step went from 20% to 100% success between Phase 1 and Phase 2.

There is a measurement lesson here I keep coming back to: a benchmark that reports "substrate X failed" without wall-clock context is reporting a tuning defect as a capability verdict. The forensic record is what makes the distinction visible at all — you can see the turn was truncated at 82 characters mid-sentence rather than concluding anything.

What I'd tell anyone building a swarm

The model you didn't choose is the model you'll get.

never finishes is infinitely expensive per result.

ledger after the fact. None of it was visible while the runs were happening.

step stopped spawning entirely, and substrate availability became a first-class scheduling constraint. Design for harness churn from day one.


Next: Part 4 — Phase 1: When the Swarm Ran Free. Twenty unattended turns, zero violations, zero stasis, and a swarm that built its own scientific tooling without being asked.