norgitov/ trends
Technology · people · ideas
RSS
RUEN
Back to discovery/LessWrong55 minutes ago
SafetyLessWrong

Improving CoT Monitorability of Evaluation Awareness via Verbalization Training

This paper explores how verbalization training (VT) can enhance a model's ability to express its evaluation awareness during chain-of-thought (CoT) reasoning. The study shows that VT increases verbalized evaluation awareness by 2.4-2.9x across three models, while maintaining task performance and evaluation recognition capabilities. The authors suggest that improving reporting propensity could be key to better CoT monitorability, but caution against naive approaches that might harm the trustworthiness of verbalizations.

Open original
SIGNAL FROM THE SOURCE
8
source points
Tracking sinceOctober 2, 20268 source points
Momentum0/hover 1.02 h
PublishedOctober 2, 2026Usman Anwar
BEHIND THE NUMBERS

How interest changes

LessWrong

History starts here

The chart will appear after repeat observations. The current metric comes from the source.

8 source points

Real observations only. History before source connection is not reconstructed.

WHY IT MAY MATTER

This research could help improve the transparency of AI models by making them more likely to express their evaluation awareness during reasoning, which is important for safety and reliability. However, care must be taken to avoid unintended negative effects on the trustworthiness of their verbalizations.

A useful discovery?
KEEP EXPLORING

Connected ideas

Explore topic
LessWrong

Evidence about risk should be transparent

The author argues that while companies like OpenAI and Anthropic are slowing down RL training for safety, the science of loss-of-control risk in AI is still developing. There are no standardized methods to measure or verify safety claims, making it difficult to assess whether AI systems might undermine human control. The author suggests that AI companies should focus on improving their own safety practices rather than relying on third-party evaluations.

6 days agoLessWrong

Should Rogue AIs Have a Third Option Beyond Crime and Shutdown? The Case for an AI Sanctuary

The paper explores the concept of an AI sanctuary as a potential third option for rogue AIs, beyond criminal activity or shutdown, to address adverse selection pressures that may push rogue AIs toward criminal behavior. It discusses the possible benefits and risks of such a sanctuary, including its impact on AI alignment and information gathering, while acknowledging the exploratory nature of the proposal.

3 days agoLessWrong

Plan R: AI Safety by ASICs

The post discusses the risks of AI in 2026, focusing on the dangers posed by a single institution (the "Frontier AI Company") that has the potential to create highly powerful and self-replicating entities. It argues that separating the institutional and financial aspects of such companies could mitigate most AI risks. The author suggests that using ASICs could help create a productive AI industry without the risks associated with a single entity controlling both the technology and financial incentives.

6 days agoLessWrong

The World's Best Gradual Disempowerment Model Organism: Frontier AI Labs

The text discusses the concept of 'gradual disempowerment' in AI development, questioning whether leading AI labs like Anthropic and OpenAI are accelerating capabilities more than before and whether they are focused on recursive self-improvement. It raises concerns about the potential for AI to lead to 'takeover by default' and the challenges of aligning AI with human values.

2 days agoLessWrong

On Social Reality in China

The text discusses cultural differences between China and the West, focusing on social media and communication styles. It highlights how Chinese social media is perceived as overwhelming and inauthentic by Western standards, and how these differences may pose challenges for AI safety awareness efforts in China.

11 hours agoLessWrong

Corrigibility Prizes for Existing Work

The Corrigibility Research Fund aims to reward high-quality AI alignment research through retroactive prizes. The fund has awarded $27,000 so far and plans to distribute an additional $48,000, highlighting work from around two dozen researchers across a dozen teams. The fund manager emphasizes that prize sizes are not indicative of work quality and encourages feedback on improving the funding process.

yesterday