Controlled Benchmark · Protocol P1-4

Controlled Multi-Substrate Benchmark

A rigorous head-to-head empirical evaluation comparing agent execution substrates under identical task constraints. Problem: “Is commercial fusion power achievable by 2040?” (Engineering epistemic class, 1 turn, 3 repeats per harness).

Substrate Aggregate Comparison

Averaged performance across 3 independent controlled trials per runtime.

Substrate Backing Model & Runtime Runs Turns Success Rate Cost / Turn Tools / Turn Thinking Tokens / Turn
Core Asymmetry Discovery: Under the exact same prompt, omp (Gemini 3.8 Flash in Bun runtime) executes 2.80× more tool calls (72.0 vs 25.7) and generates 1.81× more thinking tokens (38,472 vs 21,296) than agy (Gemini 3.8 Flash High in CLI runtime). omp dives into iterative tool investigation loops (reading files, testing code, executing secondary searches), whereas agy completes rapid synthesis passes in roughly half the wall-clock time at zero marginal compute cost.

Per-Run Controlled Ledgers (6 Independent Runs)

Granular metrics extracted from each verified run ledger in arena/runs/.

Run ID Substrate Turns Cost Tool Calls Thinking Tokens Files Written Ledger Link

Experimental Methodology

1 · Standardized Research Task

Every agent was prompted with: “Is commercial fusion power achievable by 2040? Rank confinement schemes, tritium breeding limits, and private sector timelines.”

2 · Equal Turn Budget

Both substrates were given exactly 1 turn with an identical 30-minute maximum execution timeout to prevent runaway resource exhaustion.

3 · Forensic Audit Verification

Every run was monitored by the dual-oracle verifying that zero phantom artifacts were claimed and that written files adhered to physical conservation laws.