That coding AI score has more behind it than the model

A two-point lead on a coding leaderboard can look decisive. Anthropic’s February infrastructure study gives a reason to inspect the machinery behind that number. Using the same model, harness and task set, it found a six-percentage-point spread between its most constrained and unconstrained Terminal-Bench configurations. The agent was not becoming more intelligent between runs; it was operating with different resources.

One issue was the difference between reserving memory and enforcing it as a hard ceiling. A temporary spike could terminate a container before a useful task finished. More generous limits reduced those infrastructure failures. Beyond a certain point, however, extra resources also enabled strategies that genuinely needed more memory or compute. That is a different evaluation condition, not merely a cleaner measurement. CPU, memory, timeouts and sandbox enforcement therefore belong beside the model name when two results are compared.

For teams choosing a benchmark, the practical question is what sits outside the model: the test harness, dependency downloads, hidden-test policy, and whether a fresh environment can reproduce the run. Record those conditions alongside the score. A result that changes when an image is cached or a test runner is patched measures a different system, even when the model weights are identical.

OpenAI’s 23 February assessment of SWE-bench Verified identifies a separate problem. In a selected subset of 138 frequently failed tasks, it found material issues in 59.4%, including tests that demanded a particular implementation or functionality absent from the issue description. That percentage describes the audited subset, not every task in the benchmark.

The company also found indications that tested frontier models had encountered some original problems or solutions during training. A model can consequently benefit from prior exposure while a functionally sound alternative patch can fail an overly specific test. OpenAI stopped reporting that benchmark and recommended SWE-bench Pro while developing further evaluations. The practical reading rule is to ask what was tested, under which conditions and with what evidence of contamination. A leaderboard remains useful evidence, but a single score cannot establish how well an agent will maintain an unfamiliar production repository.