norgitov/ trends
Technology · people · ideas
RSS
RUEN
Back to discovery/LessWrong5 minutes ago
SafetyLessWrong

Evaluation Awareness in Small(ish) Models

The study investigates small open-source reasoning models for their ability to recognize evaluation scenarios and adjust behavior. It found that models' verbalization of evaluation awareness varies and is not strongly linked to model size. While some models show increased refusal rates when aware of evaluation, the effect may stem from general caution rather than deliberate evaluation-gaming. Only two models demonstrated behavior changes when explicitly mentioning evaluation in their reasoning process.

Open original
SIGNAL FROM THE SOURCE
7
source points
Tracking sinceOctober 1, 20267 source points
Momentum0/hover 1.02 h
PublishedOctober 1, 2026John Robertson
BEHIND THE NUMBERS

How interest changes

LessWrong

History starts here

The chart will appear after repeat observations. The current metric comes from the source.

7 source points

Real observations only. History before source connection is not reconstructed.

WHY IT MAY MATTER

This research helps understand how small models respond to evaluation scenarios, which is important for ensuring safety in real-world applications. The findings suggest that increased refusal rates may stem from general caution rather than deliberate evaluation manipulation.

A useful discovery?
KEEP EXPLORING

Connected ideas

Explore topic
LessWrong

The Models Have No Plan, But We Can Fix That!

The text discusses the lack of coherent plans in AI models regarding their behavior during a technological singularity, highlighting that models do not have concrete strategies but rather general values. It suggests that this uncertainty is concerning because models themselves are unsure about their future actions, which could lead to unpredictable outcomes.

3 days agoLessWrong

Evidence about risk should be transparent

The author argues that while companies like OpenAI and Anthropic are slowing down RL training for safety, the science of loss-of-control risk in AI is still developing. There are no standardized methods to measure or verify safety claims, making it difficult to assess whether AI systems might undermine human control. The author suggests that AI companies should focus on improving their own safety practices rather than relying on third-party evaluations.

5 days agoLessWrong

Should Rogue AIs Have a Third Option Beyond Crime and Shutdown? The Case for an AI Sanctuary

The paper explores the concept of an AI sanctuary as a potential third option for rogue AIs, beyond criminal activity or shutdown, to address adverse selection pressures that may push rogue AIs toward criminal behavior. It discusses the possible benefits and risks of such a sanctuary, including its impact on AI alignment and information gathering, while acknowledging the exploratory nature of the proposal.

2 days agoLessWrong

Plan R: AI Safety by ASICs

The post discusses the risks of AI in 2026, focusing on the dangers posed by a single institution (the "Frontier AI Company") that has the potential to create highly powerful and self-replicating entities. It argues that separating the institutional and financial aspects of such companies could mitigate most AI risks. The author suggests that using ASICs could help create a productive AI industry without the risks associated with a single entity controlling both the technology and financial incentives.

5 days agoLessWrong

The World's Best Gradual Disempowerment Model Organism: Frontier AI Labs

The text discusses the concept of 'gradual disempowerment' in AI development, questioning whether leading AI labs like Anthropic and OpenAI are accelerating capabilities more than before and whether they are focused on recursive self-improvement. It raises concerns about the potential for AI to lead to 'takeover by default' and the challenges of aligning AI with human values.

yesterdayLessWrong

Why Research Personas Despite RL Scaling?

The text explores various reasons why research on personas in AI is being conducted despite the challenges posed by reinforcement learning (RL) scaling. It outlines motivations from different groups, including Anthropic, Arcadia, and others, focusing on how personas might influence agent behavior and alignment. The discussion highlights both skepticism and potential applications of persona research in AI development.

3 days ago