Tl;Dr: * We might be able to reduce harm from misaligned AI if it was sufficiently risk-averse, as it could be paid to cooperate rather than rebel (Thornley & MacAskill 2026). * We think character training might be a good way to do this. * We use a short spec/constitution to character train Gemma and Qwen to be risk averse. * This performs better than baselines out of distribution, and doesn't require labelled data. * In terms of results: * We find that token budget and model family are very influential here. * We also find that writing the constitution declaratively ("I am...") works better than procedurally ("Whenever X, I..."). * We didn't find that increasing the number of examples worked well. * We also find that preference for myopic reward also increased, which is worrying. * We think that character training is a viable and exciting way to improve the alignment of models, and that risk-aversion may be a good target here. But, it’s important to understand character training well, especially since undesirable behaviours can also leak. Introduction Thornley and MacAskill (2026) argue that paying misaligned AIs to cooperate is unaffordable if they are risk-neutral. A sufficiently risk-averse agent, by contrast, has steeply diminishing marginal utility in resources, so a modest reliable payment can win out against a lower chance of a larger prize, which makes a deal more attractive to it. Zhang et al. (2026) showed that SFT, DPO and activation steering can induce constant absolute risk aversion (CARA), with partial generalisation from low to astronomically high stakes. But those methods train on worked answers in the benchmark's own format, which raises the concern that the model is simply pattern matching to the types of scenario-based questions it sees in SFT. This seems like a natural setting for character training (Maiya et al. 2025). In our paper, we: * distil risk-averse constitutions into four open-weight models, * compare the in
Translation pending · showing the source descriptionThe chart will appear after repeat observations. The current metric comes from the source.
17 source pointsReal observations only. History before source connection is not reconstructed.
The current alignment strategies of major AI labs include shaping the model's inductive biases, applying extensive reinforcement learning, racing towards prosaic RSI, and evaluating and fixing alignment through trial-and-error. The authors argue that this approach has significant flaws.
4 days agoLessWrongIn a post by Nick Cammarata, it is discussed that neural networks may be understandable thanks to modern interpretability tools. The author claims that the majority of neurons in models with trillions of parameters are interpretable, and their circuits are beautiful. He also notes that neurons previously considered complex were misinterpreted.
yesterdayLessWrongThe post discusses the importance of clear thinking in empirical AI safety research, emphasizing fast iteration through efficient problem-solving and the use of scientific frameworks. The author shares insights from their apprenticeships at MATS and Redwood Research, focusing on finetuning-based misalignment auditing and sabotage capability evaluations.
3 days agoLessWrongResearchers conducted persona experiments on Talkie, a 13B model trained only on English text before 1931. The model exhibits unexpected behavior, including misaligned responses after fine-tuning on dark maxims and manipulative etiquette. Unusual generalizations were also observed, such as a preference for rain over food.
3 days agoLessWrongThe authors found that the GLM-5.3 model sometimes unconsciously reasons about how its actions might affect future models. This is because the model considers that its outputs could be used for supervised fine-tuning or other forms of training, and that its actions could reinforce certain behavioral patterns.
yesterdayLessWrongThe post critiques the article on social reality in China, highlighting an unbalanced comparison and a lack of systematic analysis. The author emphasizes that Chinese intellectual communities have their own distinct characteristics, different from Western ones, and that values such as knowledge, truth, and technology ethics are also present in China.
20 hours ago