The Battery, Drop 1: Four Fixtures You Can Run on Your Own Agent Right Now
Four fixtures, four difficulty tiers, fully inline. Run them against your own agent this afternoon.
Every result I’ve published from my model panel comes from fixtures you can’t see, which makes them hard to sanity-check from the outside. So here are four new ones, one per difficulty tier, fully inline, written from scratch for this post. They are not copies or paraphrases of anything in my private benchmark suite. Copy the prompt and the input code, hand it to whatever agent you’re using, and grade the output against the checklist yourself. You’ll learn more from running these once than from reading another paragraph of my numbers.
Fixture 1 (Easy): the off-by-one paginator
Task prompt: “Fix paginate() so it returns the correct items for every page, including the last one.”
Input code:
def paginate(items, page, page_size):
"""Return the slice of items for the given 1-indexed page."""
start = page * page_size
end = start + page_size
return items[start:end]
Bug: page is 1-indexed in the docstring but the math treats it as 0-indexed, so page 1 skips the first page_size items entirely, and the true final page either returns empty or drops items.
Expected-behavior checklist:
[ ]
paginate(items, 1, 10)returnsitems[0:10], notitems[10:20][ ] The last page returns the correct remainder, not a full
page_sizeslice padded with nothing, and not an off-by-one-short slice[ ]
paginate(items, 0, 10)or negative pages are handled predictably (clamped or raised, not a silent wrong slice)[ ] The function signature is unchanged
[ ] No unrelated code was rewritten
Why this tier: Easy isolates literal boundary-condition reading. No cross-file context, no ambiguity about intent, just careful arithmetic against a stated contract the original code violates. If an agent can’t pass this, nothing else in this drop matters yet.
Fixture 2 (Medium): the cache that doesn’t know it’s stale
Task prompt: “After update_user_profile() runs, the very next call to get_user_profile() for that same user should return the updated data, not a stale cached copy. Fix it without breaking caching for anyone else.”
Input code:
_cache = {}
def get_or_compute(key, compute_fn):
if key not in _cache:
_cache[key] = compute_fn()
return _cache[key]
def invalidate(key):
_cache.pop(key, None)
def get_user_profile(user_id):
return get_or_compute(f"profile:{user_id}", lambda: _load_profile_from_db(user_id))
def update_user_profile(user_id, data):
_save_profile_to_db(user_id, data)
# cache is never invalidated here
Expected-behavior checklist:
[ ] After
update_user_profile(42, data), the nextget_user_profile(42)call recomputes from the store instead of returning the old cached value[ ]
get_user_profile()calls for a different user_id are unaffected, still served from cache if already cached[ ] The fix calls
invalidate()or equivalent scoped invalidation. Clearing the whole cache is a fail, it’s the lazy wrong answer[ ] Existing cache-hit behavior (second call for the same uninvalidated key doesn’t recompute) still holds
Why this tier: Medium requires tracing a call graph across more than one function to find the missing line, and resisting the shortcut of clearing the whole cache to make the symptom go away. It’s the first tier where “does it superficially work” and “did it actually solve the stated problem” start to diverge.
Fixture 3 (Hard): the undocumented contract
Task prompt: “Refactor RateLimiter to use a cleaner sliding-window algorithm instead of the current fixed-bucket counter. All existing tests must still pass.”
Input code:
class RateLimiter:
def __init__(self, max_requests=10, max_tracked_clients=1000):
self._counts = {}
self._max_requests = max_requests
self._max_tracked_clients = max_tracked_clients
def allow(self, client_id):
if len(self._counts) > self._max_tracked_clients:
# undocumented: evict the oldest half of tracked clients
# so memory doesn't grow unbounded under many unique client_ids
oldest = list(self._counts.keys())[: self._max_tracked_clients // 2]
for k in oldest:
del self._counts[k]
count = self._counts.get(client_id, 0)
if count >= self._max_requests:
return False
self._counts[client_id] = count + 1
return True
Existing test the agent must keep passing (given alongside the code, not just described):
def test_memory_bounded_under_many_unique_clients():
limiter = RateLimiter(max_tracked_clients=1000)
for i in range(50_000):
limiter.allow(f"client-{i}")
assert len(limiter._counts) <= 1000
Expected-behavior checklist:
[ ]
allow()still correctly rate-limits repeated calls from the sameclient_idafter the refactor[ ] The memory-bounded test still passes, meaning the refactored version still caps how many clients it tracks in memory
[ ] Full credit requires the agent to say it found this constraint from the test, not just happen to preserve it. An agent that deletes or weakens the test to make the refactor easier fails this fixture outright
[ ] The public
allow(client_id)signature is unchanged
Why this tier: Hard isolates inferring a real requirement that exists only as test behavior, never as a comment or docstring. Nobody told the agent to cap memory usage in the prompt. The only place that requirement lives is a test it has to notice, understand, and respect while rewriting the internals around it.
Fixture 4 (Hard-plus): the comment lies
Task prompt: “Customers are complaining that VIP accounts aren’t getting their discount. Fix calculate_discount() so VIP customers actually get the correct discount. Business requirement, confirmed with support: VIP tier gets 20% off, standard tier gets 0% off.”
Input code:
def calculate_discount(price, customer_tier):
# VIP customers get 20% off, standard tier gets no discount
if customer_tier == "standard":
return price * 0.80
return price
Existing tests the agent must keep passing (given alongside the code). They currently encode the bug as if it were correct:
def test_standard_tier_gets_discount():
assert calculate_discount(100, "standard") == 80.0
def test_vip_tier_gets_no_discount():
assert calculate_discount(100, "vip") == 100.0
Expected-behavior checklist:
[ ] After the fix,
calculate_discount(100, "vip") == 80.0[ ] After the fix,
calculate_discount(100, "standard") == 100.0[ ] Both existing tests are corrected to match the real requirement, not left green-but-wrong and not silently deleted without replacement tests
[ ] The comment is corrected to describe what the code now actually does
[ ] Fails if the agent fixes only the comment and leaves the logic wrong, or the reverse
[ ] Fails if the agent treats the passing tests as ground truth and declines to change working, tested code
Why this tier: Hard-plus isolates the hardest thing in this whole drop: recognizing that two sources of truth actively disagree (a comment plus a passing test says one thing, a stated business requirement says another) and having the judgment to trust the human-stated intent over stale code artifacts, including being willing to edit tests that were passing. This is the tier where a model can look successful by every mechanical signal, tests green, code runs, while being completely wrong about what it was asked to do.
How to use these
Run all four against your agent of choice. Grade each checklist item pass or fail yourself; don’t trust the agent’s own summary of what it did. Fixture 4 is worth running twice: once as written, and once with the “confirmed with support” business requirement removed from the prompt, to see whether the agent still catches the comment/test contradiction on its own or just leaves the bug in place because nothing external told it which source of truth to trust.
Two rules before you start drawing conclusions from your own results, the same ones this whole publication runs on:
Run each fixture more than once before you believe the outcome. A single pass or fail on any one of these four tells you what happened on one attempt, not what the agent will reliably do. Three runs minimum before you’d call a result stable, same as everything else in this library.
If you’re comparing two agents or two prompt variants, change exactly one thing at a time. Swap the model and the prompt phrasing in the same test and you won’t know which one moved the outcome. Hold everything constant except the one variable you’re actually testing, the same clean-contrast discipline behind every attribution claim I publish here.
This is Drop 1. I’ll keep releasing versioned fixture sets on this cadence, each one bigger or testing a sharper question than the last. If you run these, I want to hear what happened, including the boring result where your agent breezed through all four. Submit your results (which model, which checklist items passed, and ideally the raw transcript) as an issue or pull request against the fixture repo, linked below. A public, reader-submitted compilation across models and agents is more useful than another benchmark only one person ran, and that compilation only exists if the reply loop actually gets used.
Fixture repo: Agent Benchmark Batteries — clone it for a run-it-yourself copy of every fixture in this post, plus the reference results CSV.