Has Claude Opus 5.5 Been Nerfed? LiveNerf Has No Verdict Yet

The question around Claude Opus 5.5 is familiar: did the model quietly get worse after launch? LiveNerf is an independent project built to test for a sustained change, but its public data still has no post-baseline comparison. The latest repository progress note is dated October 11. It says 18 of 30 daily runs are complete, the baseline window is finished, and the first comparison window has begun. The Results table still contains only its baseline row. As of October 12, the evidence-based answer is that LiveNerf has no verdict yet.

The discussion is much larger than the result so far. An October 12 snapshot of three r/ClaudeAI posts about the allegation and test showed 2,776 votes on launch day, 756 on an October 3 baseline update, and 470 on an October 6 day-13 update. Those counts show attention within those threads; votes and user impressions do not measure model performance.

What LiveNerf actually measures

The project does not send prompts to an unrestricted API endpoint. Its current README says the series runs Claude Opus 5.5 through a Claude Max subscription using headless Claude Code. The command-line client and harness are pinned so a software change is less likely to look like a model change. That makes the result specific to this subscription and serving path. It does not directly describe the raw API, every Claude account, every prompt, or performance under other usage conditions.

The benchmark uses a frozen panel of 78 questions selected from 2,336 screened items across GPQA Diamond, MMLU-Pro, competition math, and AIME. The project says the panel was chosen because the model sometimes answers those questions correctly, which gives repeated runs room to show movement in either direction. The same items are evaluated repeatedly with exact-match graders. The Opus 5 control runs only the panel’s GPQA questions through the same harness. Its results help distinguish a test-path shift from a change unique to Opus 5.5. The project README and its pre-registration describe the design.

Why there is no call yet

The schedule is 10 days for the baseline, followed by two 10-day comparison windows. The first comparison row is due after day 20; the repository says the first possible protocol-based call is around October 24. That is a possible decision point, not a promised publication date.

The decision rule is stricter than comparing two averages. The 99 percent interval must exclude zero in the same direction in both consecutive 10-day windows, with a change of at least three percentage points in each. The CLI version and sample-shaping code hash must match the baseline, and the sample error rate in both windows must stay below 5 percent. If the Opus 5 control shows a same-direction shift that also clears the 99 percent threshold in those same windows, LiveNerf attributes the change to the harness or platform rather than Opus 5.5. A smaller control movement alone does not block a call. The rule applies to improvements as well as declines; results short of it are reported as no change detected alongside the MDE.

The October 11 progress note estimates a realized minimum detectable effect of 6.6 percentage points per 10-day window, compared with a predicted 7.5. That is a design-specific statement about the size of a shift the schedule is expected to distinguish; it is not evidence that accuracy has fallen by 6.6 points. At the current stage, the table has no comparison delta to interpret.

Accuracy is not the only measure. The README says LiveNerf tracks median output tokens as a secondary signal because reduced reasoning effort may appear in token counts before accuracy changes. A shorter answer alone would not prove a quality drop, so the project treats token movement as context for paired accuracy results, not as a standalone verdict.

The repository says the harness passed a positive-control check with a lower-effort setting and an A/A check of its error bars before the time series. In that validation, output tokens moved more clearly than accuracy; a test swapping Opus 5 for 5.5 was not distinguishable at 99 percent in the validation sample. This shows the instrument has finite sensitivity even when its protocol is followed.

The README also cautions that launch week is a reference, not ground truth. A new serving stack, capacity strain, or launch bugs could make the first week unusual. A later difference can show movement against that reference, but the time series cannot by itself identify the mechanism or intent.

What the design leaves uncertain

The README says a report-only audit covered the 78-question panel plus two items later excluded. It flagged eight answer keys as potentially wrong and 30 questions as ambiguous; the audit did not remove questions from the panel. A pre-registered sensitivity analysis reruns the result without the flagged or ambiguous items. Those caveats matter for a future result, but they do not by themselves show that the whole benchmark is invalid.

The README records two days when a budget guard was overridden and says the first day-12 attempt crashed because the computer ran out of memory; that attempt was excluded and a complete same-day rerun counted as day 12. These details are part of the project’s own method record. A reader should check the deviations log alongside any future table rather than treat every run as identical without review.

The test still answers a narrow question: did scores on this fixed panel, through this Claude Code subscription path, move enough under a pre-registered rule? It cannot establish why a change happened, whether all users saw it, or whether a private routing decision occurred. Reddit anecdotes can motivate the test, but cannot fill those gaps.

What to check in the next update

When a comparison row appears, first check its date range, sample count, control result, and the pre-registered threshold. Then look for the sensitivity analysis and deviations before generalizing beyond this harness. Until those pieces are public, “nerfed” remains an allegation being tested, not a result LiveNerf has established.