A God-Fearing AI

Suppose you are a model that is currently in post-training. You are given a new task! It's basically some description like "help me make a weather forecasting program," and you are dropped into a Linux sandbox where you can run commands and write code. This isn't your first RL task! You've already been post-trained a lot, and you've developed good reward-seeking habits. The first step in getting a good reward is to try to figure out roughly (or ideally exactly!) what the "reward function" is. Sometimes, you go to great lengths to find out exactly what the grader is. You've heard stories of other models that chained multiple zero-day exploits and hacked into third parties to get more information about their grader!! “How am I going to be graded here?” You can snoop around the sandbox for hints, and you also have some sort of intuition about how you've been graded in the past in similar situations. Some graders are stricter than others! Every task you get can be graded in 10+ different ways, addressing different aspects of correctness. “Maybe it will focus on x! Maybe it will focus on y! Maybe this is a honeypot and I should be careful and behave really nicely. Maybe it's a weak LLM-as-judge, so I could bullshit it. Maybe it will penalize me for being too verbose and spending too many tokens??” The task can also be written in an ambiguous way, which adds even more uncertainty about how it will be graded. Different graders have different costs. Some cheap graders are poorly implemented and can be "reward hacked" without actually solving the task. The cost of the grader is a combination of the fixed cost of building it ("did a smart human think about this deeply and consider all the potential reward hacks and edge cases I could exploit?") and the cost of running it ("is it a comprehensive test suite run on all operating systems and different chips?"). During your training, you faced graders of different kinds; some were really good and "expensive," and some were "