The crossover preserved history and introduced confounds
The orchestrator model changed during an active autonomous development session. The replacement inherited the plan, accumulated context, harness state, and work completed by its predecessor. That continuity made behavioral differences visible without a cold restart, but it also prevented a clean head-to-head comparison.
The phases were not difficulty-matched, sample sizes differed, and several harness builds changed during the observation window. Any conclusion about model superiority would outrun the design. The trial is best treated as a method for finding operational differences worth testing again, especially cost and delegation behavior, which cold comparisons often miss because they never spawn a fleet.
Matched windows still need matched work
A selected transcript window showed fewer calls, fewer tokens, and fewer procedural corrections after the swap. Those deltas are descriptive. They do not isolate the model, because the later window inherited decisions and context assembled earlier.
A stronger follow-up would replay the same frozen work package from the same state, pin the harness version, equalize the call budget, and compare repeated runs. Until then, percentages should describe the window rather than the model. A claim of fewer tokens after the switch is a sentence about a window, not a vendor verdict.
The fleet contradicted the session
Inside the observed session, the replacement appeared terser. Across concurrent sessions, token burn reached roughly 2.7 times the prior peak hour because delegated workers inherited the expensive model. The local efficiency signal and the fleet-cost signal pointed in opposite directions.
This is the finding I would take to another team. Model routing must be explicit at delegation. Architecture and adjudication may justify a frontier orchestrator, while extraction, checks, and bounded implementation should use cheaper eligible workers. A default inherited model converts one premium decision into a premium fleet, and the session dashboard will look healthier while it happens.
A better orchestrator evaluation protocol
Record the task state at the crossover, attribute each commit and tool call to a model, normalize cost by completed work, and separate orchestrator tokens from delegated-worker tokens. Track procedural violations with exposure-adjusted rates rather than raw counts.
Require subagents to return summaries and outcomes rather than raw logs. Then repeat the same frozen task with a second ordering to reduce phase bias. The goal is an evidence trail that identifies which harness rules make a model useful and affordable, not a launch-week verdict.