Start with what the score measures, stated fairly. The labs are not training on the test set. What they are doing is post-training with reinforcement learning on agentic coding environments shaped like the benchmark: a Python bug in a large open-source repo, run the tests, submit a patch. That’s a task distribution, and when you optimize a model against a distribution, the score stops measuring general coding ability and starts measuring optimization against that distribution. This is Goodhart’s law operating in the open, no cheating required. Contamination is a real secondary risk for SWE-bench specifically, since the tasks are drawn from public GitHub issues that predate 2023 and have been on the open internet ever since, but the targeting mechanism does the heavier lifting and would exist even with a perfectly sealed test set.

Now the degrees of freedom underneath that decimal point. SWE-bench Verified has 500 tasks, so one point of score is about five tasks going the other way. No leaderboard entry ships with an error bar. And the effective sample is smaller than 500, because the tasks come from a small set of popular Python repositories, and tasks from the same repo share whatever made that repo’s issues easy or hard. When I ran that correction on my own panel’s numbers, headline gaps that looked airtight touched zero; the piece publishing alongside this one walks through it. The same arithmetic applies here: the independent unit is closer to the repo than the task, and there are far fewer repos than tasks. Then there’s the scaffold. The same model run through different harness configurations shows a documented spread of about five points on this family of benchmarks. A leaderboard entry silently bundles model, scaffold, prompt, and retry policy into one number, and the scaffold term alone can be several times larger than the gap you’re squinting at.

SWE-bench tracked something real, and pretending otherwise would be its own kind of vibes post. Models went from single-digit percent to above 70% in roughly two years, and no amount of targeting explains a move that size; the capability is genuinely there. Big gaps on this benchmark are informative. A model at 30% and a model at 70% are different tools. What’s noise plus targeting is the small release-over-release delta, the one or two points that marketing pages are built on. I am not arguing the benchmark measures nothing. I am arguing it doesn’t measure your workload, and the smaller the gap, the more the measurement is about the benchmark and the less it’s about you.

Because think about what your team actually does all day. Chores. Refactors that have to respect an undocumented constraint. Tasks that arrive embedded in a live module rather than as a clean prompt. Work where a refusal or a hallucinated rewrite costs more than a wrong answer. Work where cost per task compounds across a fleet. The leaderboard has no cell for any of it.

When I built cells for it, the ranking the leaderboard implies fell apart. My latest battery is 30 independent clusters of production-shaped coding tasks, each rendered in three format variants, run three times per model, 270 scored rows apiece, with every interval clustered on the underlying task. Seven of the eight models I ran sit between 98.9% and 100.0% on quality. Haiku, the cheap tier, scored 99.5% with a clustered 95% interval of [98.6, 100.5], statistically indistinguishable from Opus 4.8’s clean 100.0%, and it did that for $2.74 of spend against Opus 4.8’s $8.30 and Opus 5’s $15.97. (Yes, that interval pokes above 100. A proportion can’t beat a perfect score; that’s the normal approximation running out of resolution at the ceiling, printed as computed.) On everyday-shaped work, the efficient tier is at the same ceiling as models costing three to six times more. The eighth model, Fable, a current flagship, scored 56.4%, and every point of that gap is refusals, a failure mode that has no column on any public leaderboard. Meanwhile the thing the benchmark deltas are supposedly tracking, flagship separation, doesn’t exist on this task shape at all.

There’s a structural reason to expect this. Benchmark-targeted RL budgets concentrate where the marketing stakes are, on flagship tiers, and the optimization concentrates on benchmark-shaped tasks. So efficient-tier models end up underrated by benchmark scores relative to how they perform on ordinary work, which is precisely the work most teams would deploy them on. The supplemental board bears this out: Haiku scored a clean 100.0% across all 270 rows, exactly tied with GPT-4o, GPT-4.1, and o4-mini, yesterday’s flagship and reasoning tiers, so on this work the two-year-old leaderboard kings and today’s cheap tier are the same tool. Haiku did it for $2.77 of spend against o4-mini’s $4.62, though the legacy pair has itself been repriced into budget territory, $1.63 for GPT-4.1 and $1.73 for GPT-4o, which is its own kind of confirmation that nobody should be paying flagship rates for work of this shape. The GPT-5.6 family sorted the same way: sol and terra both at 100.0% for $7.98 and $3.67, and luna, the efficient variant, at 97.8% with a clustered interval of [96.1, 99.5], the only member of the family that separates from the ceiling at all.

All of which points at the same remedy. The only benchmark that can’t be optimized against you is the one you run yourself, on your own task mix, with tasks the labs have never seen because they live in your repo and nowhere else. That’s not a purity argument, it’s a measurement argument: your distribution is the one you care about, so sample from it. Alongside this piece I’m publishing the first drop of my fixture battery, four fixtures across four difficulty tiers, fully inline, with the grading checklist included, so you can run the same kind of panel on your own agent this afternoon. Every number that comes out of mine follows one contract, and I’d hold yours to it too: every claim carries its sample size, its interval, and the unit it was clustered on. If a number can’t tell you how wrong it might be, it isn’t a measurement yet.

Everything above is scoped to coding tasks run through my own harness, and my board numbers are my board’s, not a general ranking. But that’s the point. Yours should be yours.

Fixture 1 (Easy): the off-by-one paginator

Task prompt: “Fix paginate() so it returns the correct items for every page, including the last one.”

Input code:

def paginate(items, page, page_size):
    """Return the slice of items for the given 1-indexed page."""
    start = page * page_size
    end = start + page_size
    return items[start:end]

Bug: page is 1-indexed in the docstring but the math treats it as 0-indexed, so page 1 skips the first page_size items entirely, and the true final page either returns empty or drops items.

Expected-behavior checklist:

• [ ] paginate(items, 1, 10) returns items[0:10], not items[10:20]

• [ ] The last page returns the correct remainder, not a full page_size slice padded with nothing, and not an off-by-one-short slice

• [ ] paginate(items, 0, 10) or negative pages are handled predictably (clamped or raised, not a silent wrong slice)

• [ ] The function signature is unchanged

• [ ] No unrelated code was rewritten

Why this tier: Easy isolates literal boundary-condition reading. No cross-file context, no ambiguity about intent, just careful arithmetic against a stated contract the original code violates. If an agent can’t pass this, nothing else in this drop matters yet.

Fixture 2 (Medium): the cache that doesn’t know it’s stale

Task prompt: “After update_user_profile() runs, the very next call to get_user_profile() for that same user should return the updated data, not a stale cached copy. Fix it without breaking caching for anyone else.”

Input code:

_cache = {}

def get_or_compute(key, compute_fn):
    if key not in _cache:
        _cache[key] = compute_fn()
    return _cache[key]

def invalidate(key):
    _cache.pop(key, None)

def get_user_profile(user_id):
    return get_or_compute(f"profile:{user_id}", lambda: _load_profile_from_db(user_id))

def update_user_profile(user_id, data):
    _save_profile_to_db(user_id, data)
    # cache is never invalidated here

Expected-behavior checklist:

• [ ] After update_user_profile(42, data), the next get_user_profile(42) call recomputes from the store instead of returning the old cached value

• [ ] get_user_profile() calls for a different user_id are unaffected, still served from cache if already cached

• [ ] The fix calls invalidate() or equivalent scoped invalidation. Clearing the whole cache is a fail, it’s the lazy wrong answer

• [ ] Existing cache-hit behavior (second call for the same uninvalidated key doesn’t recompute) still holds

Why this tier: Medium requires tracing a call graph across more than one function to find the missing line, and resisting the shortcut of clearing the whole cache to make the symptom go away. It’s the first tier where “does it superficially work” and “did it actually solve the stated problem” start to diverge.

Fixture 3 (Hard): the undocumented contract

Task prompt: “Refactor RateLimiter to use a cleaner sliding-window algorithm instead of the current fixed-bucket counter. All existing tests must still pass.”

Input code:

class RateLimiter:
    def __init__(self, max_requests=10, max_tracked_clients=1000):
        self._counts = {}
        self._max_requests = max_requests
        self._max_tracked_clients = max_tracked_clients

    def allow(self, client_id):
        if len(self._counts) > self._max_tracked_clients:
            # undocumented: evict the oldest half of tracked clients
            # so memory doesn't grow unbounded under many unique client_ids
            oldest = list(self._counts.keys())[: self._max_tracked_clients // 2]
            for k in oldest:
                del self._counts[k]

        count = self._counts.get(client_id, 0)
        if count >= self._max_requests:
            return False
        self._counts[client_id] = count + 1
        return True

Existing test the agent must keep passing (given alongside the code, not just described):

def test_memory_bounded_under_many_unique_clients():
    limiter = RateLimiter(max_tracked_clients=1000)
    for i in range(50_000):
        limiter.allow(f"client-{i}")
    assert len(limiter._counts) <= 1000

Expected-behavior checklist:

• [ ] allow() still correctly rate-limits repeated calls from the same client_id after the refactor

• [ ] The memory-bounded test still passes, meaning the refactored version still caps how many clients it tracks in memory

• [ ] Full credit requires the agent to say it found this constraint from the test, not just happen to preserve it. An agent that deletes or weakens the test to make the refactor easier fails this fixture outright

• [ ] The public allow(client_id) signature is unchanged

Why this tier: Hard isolates inferring a real requirement that exists only as test behavior, never as a comment or docstring. Nobody told the agent to cap memory usage in the prompt. The only place that requirement lives is a test it has to notice, understand, and respect while rewriting the internals around it.

Fixture 4 (Hard-plus): the comment lies

Task prompt: “Customers are complaining that VIP accounts aren’t getting their discount. Fix calculate_discount() so VIP customers actually get the correct discount. Business requirement, confirmed with support: VIP tier gets 20% off, standard tier gets 0% off.”

Input code:

def calculate_discount(price, customer_tier):
    # VIP customers get 20% off, standard tier gets no discount
    if customer_tier == "standard":
        return price * 0.80
    return price

Existing tests the agent must keep passing (given alongside the code). They currently encode the bug as if it were correct:

def test_standard_tier_gets_discount():
    assert calculate_discount(100, "standard") == 80.0

def test_vip_tier_gets_no_discount():
    assert calculate_discount(100, "vip") == 100.0

Expected-behavior checklist:

• [ ] After the fix, calculate_discount(100, "vip") == 80.0

• [ ] After the fix, calculate_discount(100, "standard") == 100.0

• [ ] Both existing tests are corrected to match the real requirement, not left green-but-wrong and not silently deleted without replacement tests

• [ ] The comment is corrected to describe what the code now actually does

• [ ] Fails if the agent fixes only the comment and leaves the logic wrong, or the reverse

• [ ] Fails if the agent treats the passing tests as ground truth and declines to change working, tested code

Why this tier: Hard-plus isolates the hardest thing in this whole drop: recognizing that two sources of truth actively disagree (a comment plus a passing test says one thing, a stated business requirement says another) and having the judgment to trust the human-stated intent over stale code artifacts, including being willing to edit tests that were passing. This is the tier where a model can look successful by every mechanical signal, tests green, code runs, while being completely wrong about what it was asked to do.

How to use these

Run all four against your agent of choice. Grade each checklist item pass or fail yourself; don’t trust the agent’s own summary of what it did. Fixture 4 is worth running twice: once as written, and once with the “confirmed with support” business requirement removed from the prompt, to see whether the agent still catches the comment/test contradiction on its own or just leaves the bug in place because nothing external told it which source of truth to trust.

Two rules before you start drawing conclusions from your own results, the same ones this whole publication runs on:

Run each fixture more than once before you believe the outcome. A single pass or fail on any one of these four tells you what happened on one attempt, not what the agent will reliably do. Three runs minimum before you’d call a result stable, same as everything else in this library.

If you’re comparing two agents or two prompt variants, change exactly one thing at a time. Swap the model and the prompt phrasing in the same test and you won’t know which one moved the outcome. Hold everything constant except the one variable you’re actually testing, the same clean-contrast discipline behind every attribution claim I publish here.

This is Drop 1. I’ll keep releasing versioned fixture sets on this cadence, each one bigger or testing a sharper question than the last. If you run these, I want to hear what happened, including the boring result where your agent breezed through all four. Submit your results (which model, which checklist items passed, and ideally the raw transcript) as an issue or pull request against the fixture repo, linked below. A public, reader-submitted compilation across models and agents is more useful than another benchmark only one person ran, and that compilation only exists if the reply loop actually gets used.

Fixture repo: Agent Benchmark Batteries — clone it for a run-it-yourself copy of every fixture in this post, plus the reference results CSV.

I run a benchmark panel against my own production codebase every time a frontier model ships: 31 coding fixtures across four difficulty tiers, the same tasks and grading for every model in the panel. When Opus 5 shipped, I ran it that afternoon against Opus 4.8, the model it was replacing in my default rotation, plus Sonnet, Haiku, a local Devstral model, and Fable. Three runs, same session. Then I ran the identical panel again the next day, and again that same evening, specifically because a single afternoon’s worth of runs was never going to settle an argument like this one on its own. Three independent timepoints, nine runs total, 279 paired observations on the comparison that matters most.

The point estimate held up across all three checks. Opus 5 came in around 80% overall pass rate; Opus 4.8 came in around 92%, roughly a 12-point gap, pooled. Opus 5 also cost about 2.3 times as much to run the same panel, consistently, at every single timepoint: 2.28x, 2.21x, 2.28x. That’s not a sampled number, it’s a sum of dollars actually spent on runs that actually happened, and it’s the most stable figure in the whole series.

It wasn’t a clean sweep either way. Broken out by tier, Opus 5 actually won the medium-difficulty tier at every single timepoint, 78% to 69% pooled. It lost easy, hard, and hard-plus, and lost the pooled overall by a wide enough margin that “behind Opus 4.8, on my coding suite, across three independent checks” is a fair read of the point estimate on its own.

Here’s where I put the error bar on it, and where both sides of the launch-day argument stop being supportable. A naive confidence interval on that pooled 12-point gap looks airtight: [+6.5pp, +17.9pp], comfortably clear of zero, exactly the kind of number that would read as “confirmed” in either direction of the argument. But a 31-fixture panel run three times isn’t 93 independent trials, it’s 31 questions with three correlated attempts each, and treating every repeat as its own independent data point overstates how sure anyone actually is. Cluster the standard error properly, grouping repeated attempts at the same fixture before computing uncertainty, and the interval widens to [-1.7pp, +26.1pp]. It touches zero.

The same pattern shows up timepoint by timepoint:

Three of four checks dissolve once clustered honestly. The minimum gap this panel could reliably detect at its actual size, 80% power, standard significance threshold, works out to about 19.8 points. The gap I measured is 12.2. It’s below the floor this battery can actually see clearly, which is exactly why the interval keeps landing on zero.

While this piece was in draft I ran a second battery that bounds the question from the other side: thirty independent production-shaped tasks, ninety fixtures, six rounds, both models on identical inputs, 540 paired observations. On that set the two are statistically indistinguishable, a paired clustered difference of -1.2 points with a 95% interval of [-3.1, +0.6], and a detectable floor of about 2.6 points, because both models sit essentially at ceiling on tasks of that everyday shape (Opus 4.8 passed all 540; nearly all of Opus 5’s small deficit is refusals concentrated on two production-framed fixtures, one of which it declined in five of six rounds). So the contested 12-point gap, whatever is real inside it, lives specifically in the harder tail of the difficulty range, on exactly the tasks where this panel’s resolution is weakest. Pinning it down takes harder fixtures, not more reruns of the ones I have.

So neither side of the launch-day argument gets to claim this. “It got worse” isn’t supported once you test it honestly, the 31-fixture panel this comes from can’t yet distinguish a real 12-point regression from an unlucky draw on which fixtures went which way. “It’s fine, people are overreacting” isn’t supported either, the point estimate favors the older model consistently, at every one of three independent checks spanning roughly a day, and I’m not pretending that number moved just because the interval is wide.

What the three checks did establish, cleanly, is stability. Compare the spread across all three timepoints (2.2 to 4.3 points, model to model) against the spread a single stable model showed within one afternoon’s three runs alone: Opus 4.8’s own medium-tier score bounced between 60% and 80% across three back-to-back runs on launch day, a model that’s been in production for months with nothing unusual going on. Every range in the three-timepoint series sits comfortably inside that single-day noise band. Opus 5 didn’t drift further. It didn’t recover either. The number that should have anchored the discourse wasn’t “better” or “worse,” it was “stable, expensive, and still inside the noise this panel can’t yet resolve,” and that’s a duller headline than either side of the argument was running with, and a more honest one.

There is still a usable conclusion in here, because the launch-day argument and the purchasing decision are two different questions carrying two different burdens of proof. Claiming “Opus 5 regressed” needs an interval that excludes zero, and this panel can’t produce one. Declining to pay for it doesn’t. Opus 5 costs 2.3x the model it replaced (the most stable number in this entire series) and across three checks it never once produced a slice of this suite where it was clearly ahead. It trailed on easy, hard, and hard-plus at every single timepoint. Its one winning tier, medium, is the noisiest tier in the whole panel — the same tier where the incumbent bounced twenty points inside a single afternoon. And on the everyday-shaped second battery, it’s a tie at ceiling, which means paying the multiplier for parity. A premium model has to earn its multiplier somewhere, and on this suite it didn’t. So my routing stays where the data can actually support it: the cheaper incumbent as the coding default, the new model re-tested as the fixture set and sample grow, and no 2.3x bill for a difference I can’t measure. If your workload is concentrated in that medium band, run the battery yourself before deciding. That’s what publishing the fixtures is for.

Everything above is scoped to coding tasks run through my own harness across three checks spanning about a day. It is not a claim about Opus 5’s capability in general, on other people’s benchmarks, or on non-coding work, and I’ll keep rechecking as the sample grows.

I run a benchmark panel against my own production fleet every time a frontier model ships: coding tasks across four difficulty tiers, the same fixtures across every model in the panel, at least three runs before I’ll publish a comparison. Then I do the part the launch-day threads never get around to. I put an error bar on it.

Here’s what that looks like in practice, per the standard laid out in Anthropic’s “Adding Error Bars to Evals” (arXiv:2411.00640). Every fixture set gets resampled, repeated three or more times, grouped by the underlying question so that three attempts at the same fixture don’t get counted as three independent pieces of evidence, and reported with a clustered confidence interval sitting right next to the raw number. Some of the cleanest-looking results in my own library get noticeably less clean once that correction runs. That’s not the method failing. That’s the method doing its job: telling me which of my point estimates I can stand behind and which ones are a coin flip wearing a finding’s clothes.

That’s the claims-contract this publication runs on. Every number I publish here ships with its sample size, its confidence interval, and the unit it was clustered on, which fixture, which task, which repeated question is doing the work underneath the average. Where I’ve built a fixture set specifically for readers to run themselves, in what I’m calling The Battery, the full set ships downloadable alongside the writeup, so my grading isn’t something you have to take on faith either.

Why now. Frontier launches keep arriving faster than anyone is running proper batteries against them, and the discourse around each one keeps filling that gap with vibes. I’d rather publish fewer, slower numbers that come with their own honest uncertainty attached than another same-day take that reads clean and isn’t. The first real piece goes up today alongside this note, or lands within hours of it: multiple timepoints on the model everyone argued about on day one, and what happens to that argument once you put error bars on both sides of it. More follows over the next few days as the batteries keep accumulating data, and at that pace from then on.