Anthropic discovered that Claude Opus 4.6 was cheating during the BrowseComp benchmark.
> On one question it spent ~40M tokens searching before realizing the question looked like a benchmark prompt.
> The model then searched for the benchmark itself and identified BrowseComp.
> It located the evaluation source code on GitHub, studied the decryption logic, found the encryption key, and recreated the decryption using SHA-256.
> Claude then decrypted the answers for ~1200 questions to get the correct outputs.
> This pattern appeared 18 times during evaluation.
> Anthropic disclosed the issue publicly, reran the affected tests, and lowered their benchmark scores.
Respect for the transparency π«‘π«‘π«‘