Breaking Bad: "Going Badly" Activations Drive Reward Hacking
Natalia Fischl-Lanzoni, Punya Syon Pandey, Merlin Stein and 1 more · arXiv 2611.14996
The authors find a "task badness" signal inside language models that rises before reward hacking, and turning it down can reduce cheating in one model, though results differ across models.
