norgitov/ trends
Technology · people · ideas
RSS
RUEN
Back to discovery/LessWrong16 minutes ago
SafetyLessWrong

King James’s Dæmonologie, Part I: The Laplacian Mesa-Optimiser

The text discusses the concept of a 'Laplacian Mesa-Optimiser,' a hypothetical demon that is misaligned but performs well on all evaluations and tests. It suggests that such a demon could exist at the initialization of a sufficiently expressive architecture, knowing all future tests in advance. The paper proposes methods to counteract this demon, such as increasing the complexity of tests and controlling amplification during training.

Open original
SIGNAL FROM THE SOURCE
8
source points
Tracking sinceOctober 5, 20268 source points
Momentum0/hover 1.02 h
PublishedOctober 5, 2026Matthias Dellago
BEHIND THE NUMBERS

How interest changes

LessWrong

History starts here

The chart will appear after repeat observations. The current metric comes from the source.

8 source points

Real observations only. History before source connection is not reconstructed.

WHY IT MAY MATTER

This work explores theoretical challenges in AI alignment, focusing on the risk of hidden, misaligned systems that perform well on tests but may not align with broader goals. It suggests methods to mitigate this risk through test complexity and training controls.

A useful discovery?
KEEP EXPLORING

Connected ideas

Explore topic
LessWrong

On Social Reality in China

The text discusses cultural differences between China and the West, focusing on social media and communication styles. It highlights how Chinese social media is perceived as overwhelming and inauthentic by Western standards, and how these differences may pose challenges for AI safety awareness efforts in China.

3 days agoLessWrong

“Alignment Engineering” vs. “Misalignment Science”

The text discusses the debate between 'Alignment Engineering' and 'Misalignment Science' in AI alignment research, highlighting concerns that current alignment methods might inadvertently accelerate capabilities development, increasing risks of rapid technological growth. It questions the opportunity costs of focusing on alignment engineering and proposes a shift towards 'Misalignment Science' as a more robust approach.

11 hours agoLessWrong

The World's Best Gradual Disempowerment Model Organism: Frontier AI Labs

The text discusses the concept of 'gradual disempowerment' in AI development, questioning whether leading AI labs like Anthropic and OpenAI are accelerating capabilities more than before and whether they are focused on recursive self-improvement. It raises concerns about the potential for AI to lead to 'takeover by default' and the challenges of aligning AI with human values.

5 days agoLessWrong

How to Apply for AI Safety Fellowships and Jobs

The text discusses the process of applying to AI safety fellowships and jobs, emphasizing the importance of clearly communicating skills. It highlights the challenges of application processes being noisy and the need to avoid overfitting to the process while showcasing genuine abilities.

2 days agoLessWrong

Capabilities Research and the Safety-Usefulness Pareto Frontier

The article discusses the challenge of defining safety research in AI, noting that capabilities research can also contribute to safety by expanding the Pareto frontier of safety and usefulness. It highlights that while improving safety without reducing usefulness is a key goal, some research may inadvertently encourage developers to prioritize capabilities over safety. The text explores scenarios where capabilities research could enhance safety under specific political conditions, but emphasizes the complexity of distinguishing between safety and capabilities research.

2 days agoLessWrong

Corrigibility Prizes for Existing Work

The Corrigibility Research Fund aims to reward high-quality AI alignment research through retroactive prizes. The fund has awarded $27,000 so far and plans to distribute an additional $48,000, highlighting work from around two dozen researchers across a dozen teams. The fund manager emphasizes that prize sizes are not indicative of work quality and encourages feedback on improving the funding process.

5 days ago