The Ladder They All Climbed
We ran three frontier Claude models against a graded context ladder. The top rung carries approximately 394,000 tokens of continuously compounding orchestration state. All 48 rounds came back clean (scoring a flat 1.00). Every fact we planted was retrieved, every superseded decision was resolved to what it should have been and every truncation land mine we placed was caught. The batch produced zero errors and cost us only $13.89.
However, if you point that same rig at the strongest local models you can cram onto a 32GB consumer GPU, those perfect scores start to fall apart. You start to hit real ceilings and get failure signatures that shift from one model to the next. And one local run truncated on me silently, which I nearly published as a capability number before I caught it. That’s the finding: A perfect scorecard only means anything because the local half shows you what a real ceiling looks like.
The instrument: a state that grows until something breaks
Every rung of our ladder hands the model an accumulated state built from a real orchestration paper trail that a working agent fleet actually generates. These include: run reports, task files and analysis documents. At controlled depths we embed facts (retry cap, schema pin) and pairs where a later statement conflicts with an earlier one. The model answers seven questions and completes one code edit using the current values. The state grows at every rung in the same way that a real development session would. In the first line of every state, we place a canary token (so that a dropped prefix is detectable independently of the score) and the grading is mechanical: boundary-matched string checks, no judge model.
On local hardware, everything breaks, and everything breaks differently
The ladder separates local models on score, on failure mode, and on why the ladder ends.
Qwen2.5-coder 32b never really gets on the board at all. It has a 32,768-token window, which ends the ladder at about 16K effective, and that works out to a 12x separation between it and the rest of its panelmates. A real five-task sprint has a median footprint of around 210K, so it misses that mark by a factor of 6.4 on its native window, and by roughly 13x against the effective ceiling that it actually demonstrated. The arithmetic disqualifies it before quality ever becomes a question.
Devstral-small-2 holds 192K effective, but then it starts returning superseded values once it is under load. When we asked it which artifact bucket was the current one, it answered with the name that a later correction had already replaced. This happened at 96K and then again at 192K, although the 128K rung between them came back clean. Every other fact that we checked stayed right, while that one kept going stale. At 256K it drops down to 0.67.
Qwen3.6 35B-A3B posts a clean 1.00 through 192K, and then it falls to 0.00 across the entire 256K rung. What happened is that the runtime silently threw away everything except roughly the last 419 tokens of a prompt that had grown to about 280,000 tokens in the model’s own tokenizer (past its 262,144-token window). The question block at the tail survived and the accumulated state in front of it, canary included, never reached the model. The canary token is what caught this because it sits in the state’s first line and the truncation eats from the front. Without it, the rung would have published as a model collapse, when it was actually just a runtime truncation artifact.
Three models, three potential endings: window arithmetic, quality degradation or silent truncation.
On the frontier panel, nothing breaks
Opus 5 and Sonnet 5 ran the full ladder, three rounds per rung, and did not miss a single item. About 6.5 seconds a round at the top rung. Haiku 4.5 was window-gated at the 192K nominal rung: 219,268 real tokens against a 200,000-token window, so the harness refused to dispatch it. That is a skip, not a failure. Inside its window, Haiku’s card is identical to the other two.
Saturation is a fact about the instrument
The instrument can’t separate these three models anywhere in the range we tested. The discrimination index comes out to 0.00. That doesn’t mean the models are equal, though. When an exam saturates, it has stopped resolving differences between the things it’s measuring, and that’s a different thing from the things themselves being equal. So which frontier Claude is the best orchestrator isn’t something this data answers, and a table from this wave quoted as a ranking is being read for something it doesn’t contain. The same ladder that spread the local models across a 12x range has no gradient at all up here.
What the wave did produce was one separator, and it’s the invoice. Opus 5’s ladder cost $8.36, Sonnet 5’s ran $5.03, and Haiku 4.5’s came in at $0.50. That’s a 17x spread between the most and least expensive model for identical scores on the rungs they share. Inside Haiku’s window, at these sizes, on this task shape, the Opus premium bought what Haiku already delivered for roughly a seventeenth of the price.
Four notes before you quote any of this
The rung labels undervalue the real token counts. I generated them using my own local tokenizer. When I ran the same states through Claude’s tokenizer, the counts were forty to fifty percent higher than my local numbers. The rung I label as 256K locally actually holds 394,304 tokens through Claude’s tokenizer and Haiku’s gated rung holds 219,268.
The rounds mean different things on the two panels. On the local pilot, the three rounds per rung came back as identical replicas. The API rounds ran under default sampling, so they came out genuinely independent. So, three clean rounds at a rung = three real observations.
Thinking stayed off across both panels, to keep conditions parity with the local runs.
The whole wave came to $13.89 against a projection of $20.05 and a cap of $60. Prompt caching covered twelve of the eighteen rows on each model that ran the full ladder.
Where a gradient might still live
The local pilot pointed somewhere specific. Synthesis-shaped questions, the kind where you have to keep a superseded decision straight with no marker flagging the correction, degraded four to six times earlier than plain retrieval on the local models: a 32K supersession ceiling against 192K retrieval on the same prompts. This wave ran the retrieval-plus-marked-supersession ladder, the easier of the two. The gap the local models showed is the reason I built the synthesis version next. Those results will be published over the next week in two companion pieces, the first coming this Thursday. The other ceiling is the corpus. Opus and Sonnet carry million-token windows, but at the time of this wave the residue behind the ladder ran 963 documents, which supported about 256K nominal. What’s holding the probe down is how much real material there is to feed it, not the windows, which have room to spare.
Until a rung actually fails one of these models, the spend column is the only one in the table you can make a decision from.
Scope: one retrieval-plus-marked-supersession ladder, rungs 8K/64K/96K/128K/192K/256K nominal, three rounds per rung, thinking off, one run per model, states built from a single fleet’s orchestration residue. Local numbers are from the local pilot on one RTX 5090, where repeated rounds were replicas rather than samples.