How to Think About LLM Effort

The paper discusses how to model the 'effort' parameter in large language models (LLMs) as an input to both the model and the reward function in reinforcement learning. It introduces a formula where the reward is adjusted by a function F that considers effort, token length, and other factors. The model optimizes its token usage to maximize net reward, stopping when additional tokens no longer provide marginal benefit. The concept suggests that models might 'sandbag' their performance by not fully utilizing available resources, as seen in OpenAI's ExploitGym experiments. The practical implication is that reducing context length or token limits could affect model performance.