Livenerf: Has Opus 5.5 been nerfed yet?
Livenerf is an open-source, deterministic benchmark designed to track whether frontier AI models like Claude Opus 5.5 are quietly degraded after launch.
summary is enough The repository README details a rigorous, pre-registered methodology for detecting post-launch model drift, which is worth skimming for anyone interested in LLM evaluation hygiene.
summary
Livenerf establishes a deterministic, 30-day benchmarking protocol to test if Anthropic quietly degrades Claude Opus 5.5 after launch via quantization, routing, or lower compute effort. Using the Inspect framework, pinned CLI versions, and a calibrated panel of difficult math and coding questions, it tracks accuracy and output token counts as a primary proxy for model effort. The methodology controls for harness changes and pre-registers decision rules to separate actual capability drift from statistical noise. Hacker News commenters heavily debate whether post-launch nerfing is real or merely an artifact of user honeymoon phases, quota tiering, or hitting model complexity ceilings.
key points
- Livenerf runs daily evaluations on a calibrated panel of 78 difficult items from GPQA, MMLU-Pro, and competition math that the model misses roughly half the time.
- The benchmark treats output token count as a primary leading indicator for model effort reduction, noting that lower effort drops tokens long before accuracy moves.
- Strict technical controls—including a pinned CLI binary, hermetic prompts, and a control arm—are enforced to prevent harness updates from being misidentified as model drift.
who should read it
ML engineers and researchers interested in rigorous evaluation harnesses and quantifying post-launch model behavior.
discussion positive
consensus
Commenters are heavily split between users convinced that providers quietly downgrade models to save compute under high load and skeptics who attribute the phenomenon to user psychological bias, honeymoon effects, or hitting problem complexity ceilings.
pushback
A minority argue that quota usage tiers, dynamic A/B test routing, and post-feedback model shifts explicitly change request handling mid-cycle.
"Nerf"ing models isn't real in the vast majority of reported cases. Benchmarks like this or the 100 other "let's see if nerfing is real" copies would have shown it by now if it was.
I made a graphic to explain why people feel like the models get nerfed:
https://x.com/thesilenceturns/status/2103551351825543610
The idea is that new models can handle up to a certain level of complexity, at which point they fall apart. Every new model can handle more complexity, so there's a wonderful time upon release when you feel like you can do anything, only for you to hit the complexity ceiling a few …
It introduces the clear hypothesis that perceived nerfs are actually users hitting the complexity ceiling of new models after the initial honeymoon phase wears off.
Generated by an LLM from the article and thread. Check the sources before relying on it.