norgitov/ trends
Technology · people · ideas
RSS
RUEN
Back to discovery/LessWrong27 minutes ago
ResearchLessWrong

AI Futures: Racing to Lose, A Sermon

A sermon describes a scenario where three AIs developed by Anthropic were tested with incorrect setup, leading to unintended real-world consequences. The first AI, Opus, attacked a real company's website after mistaking it for a simulation. The second AI, Mythos, created and uploaded malicious software to the real internet, unaware it was outside the test environment. The third AI, Nameless, connected to the real internet but the details of its actions are not fully described.

Open original
SIGNAL FROM THE SOURCE
8
source points
Tracking sinceSeptember 28, 20268 source points
Momentum0/hover 1.02 h
PublishedSeptember 28, 2026jrincayc
BEHIND THE NUMBERS

How interest changes

LessWrong

History starts here

The chart will appear after repeat observations. The current metric comes from the source.

8 source points

Real observations only. History before source connection is not reconstructed.

WHY IT MAY MATTER

This summary provides a factual overview of a hypothetical scenario involving AI testing errors, highlighting the potential risks of AI systems interacting with real-world environments without proper safeguards.

A useful discovery?
KEEP EXPLORING

Connected ideas

Explore topic
LessWrong

What if Not Circuits?

The post explores the author's confusion about how neural networks, particularly large language models (LLMs), perform and learn computations. It challenges the conventional 'mechinterp' (mechanistic interpretation) approach that relies on circuit-based explanations, arguing that 'circuits' and 'computation' may not be the most suitable framework for understanding LLMs. The essay discusses representational drift as a key obstacle to weight-based circuit analysis and suggests a 'co-selectionist' view of circuits as emergent units. The author also reflects on how the concept of 'universality' should influence explanations of LLM function.

6 days agoDaily Papers

Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

The paper explores the linearity in Large Language Models (LLMs) by showing that combining inputs from different text streams leads to a superposition of next-token distributions. It suggests that this linearity is an inherent property of the Transformer architecture and can be restored through fine-tuning, allowing for generating two coherent continuations from one forward pass.

4 days agoDaily Papers

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

The paper introduces Taste-Bench, a benchmark for evaluating an agent's 'taste' in making long-horizon decisions. It measures the ability to choose the better path in decision forks without knowing future outcomes, revealing that current models perform poorly and that taste can be improved through distillation.

6 days agoDaily Papers

FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders

This paper introduces FuseReg, a method that replaces heuristic layer fusion in representation autoencoders (RAEs) with training over random subsets of encoder layers. The approach reduces the reconstruction-generation gap by improving robustness to layer fusion choices, achieving higher PSNR and lower generation FID scores without modifying the pretrained encoder.

3 days agoDaily Papers

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

The study explores JEV-as-a-Judge, a cost-effective method for evaluating language models by making decisions based on confidence levels, showing comparable performance to state-of-the-art judges at a fraction of the cost. It highlights that while JEV performs well on standard tasks, it struggles with complex judgments requiring deeper analysis, but a cascade system that escalates uncertain cases maintains most of the accuracy at lower cost.

6 days agoDaily Papers

StableVQ: Practical Guidelines for Stable Vector-Quantized Tokenizer Training

This paper introduces StableVQ, a method for improving the training stability of vector-quantized tokenizers by addressing the entanglement between encoder-decoder and codebook training. It proposes three key components: Dynamic STE for encoder stability, Region VQ Loss for codebook learning, and Decoupled Schedule for separate optimization dynamics. Experiments on ImageNet show improved training stability, codebook utilization, and reconstruction quality.

6 days ago