norgitov/ trends
Technology · people · ideas
RSS
RUEN
Back to discovery/arXiv1 hour ago
ResearcharXiv

When the Preconditioning Exponent Turns Negative: Learning-Rate Coupling and Cross-Environment Generalization

This study investigates how the preconditioning exponent in adaptive optimizers interacts with the learning rate, revealing that the exponent maximizing cross-environment accuracy decreases linearly with the logarithm of the learning rate. Negative exponents are not universally optimal but represent a high-step-size allocation regime influenced by the joint action of learning rate and preconditioning. The research highlights a model-selection conflict between source-domain validation and robustness to environmental changes.

Open original
SIGNAL FROM THE SOURCE
28
September 2026
Tracking sinceSeptember 28, 2026
Momentum—More observations needed
Discussion—No comment count provided
PublishedSeptember 28, 2026Gongyue Zhang, Honghai Liu
BEHIND THE NUMBERS

How interest changes

arXiv

The publication is the signal

This official source does not publish popularity metrics. The story is refreshed from its RSS feed.

Real observations only. History before source connection is not reconstructed.

WHY IT MAY MATTER

This study provides insights into optimizing adaptive optimizers by showing how the preconditioning exponent interacts with the learning rate, which can help in selecting better hyperparameters for robust cross-environment generalization.

A useful discovery?
KEEP EXPLORING

Connected ideas

Explore topic
Daily Papers

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

The study explores JEV-as-a-Judge, a cost-effective method for evaluating language models by making decisions based on confidence levels, showing comparable performance to state-of-the-art judges at a fraction of the cost. It highlights that while JEV performs well on standard tasks, it struggles with complex judgments requiring deeper analysis, but a cascade system that escalates uncertain cases maintains most of the accuracy at lower cost.

6 days agoDaily Papers

Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs

The paper explores the linearity in Large Language Models (LLMs) by showing that combining inputs from different text streams leads to a superposition of next-token distributions. It suggests that this linearity is an inherent property of the Transformer architecture and can be restored through fine-tuning, allowing for generating two coherent continuations from one forward pass.

4 days agoDaily Papers

The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks

The paper introduces Taste-Bench, a benchmark for evaluating an agent's 'taste' in making long-horizon decisions. It measures the ability to choose the better path in decision forks without knowing future outcomes, revealing that current models perform poorly and that taste can be improved through distillation.

6 days agoDaily Papers

FuseReg: Regularizing Layer Fusion Mitigates the Reconstruction-Generation Gap in Representation Autoencoders

This paper introduces FuseReg, a method that replaces heuristic layer fusion in representation autoencoders (RAEs) with training over random subsets of encoder layers. The approach reduces the reconstruction-generation gap by improving robustness to layer fusion choices, achieving higher PSNR and lower generation FID scores without modifying the pretrained encoder.

3 days agoLessWrong

What if Not Circuits?

The post explores the author's confusion about how neural networks, particularly large language models (LLMs), perform and learn computations. It challenges the conventional 'mechinterp' (mechanistic interpretation) approach that relies on circuit-based explanations, arguing that 'circuits' and 'computation' may not be the most suitable framework for understanding LLMs. The essay discusses representational drift as a key obstacle to weight-based circuit analysis and suggests a 'co-selectionist' view of circuits as emergent units. The author also reflects on how the concept of 'universality' should influence explanations of LLM function.

6 days agoDaily Papers

StableVQ: Practical Guidelines for Stable Vector-Quantized Tokenizer Training

This paper introduces StableVQ, a method for improving the training stability of vector-quantized tokenizers by addressing the entanglement between encoder-decoder and codebook training. It proposes three key components: Dynamic STE for encoder stability, Region VQ Loss for codebook learning, and Decoupled Schedule for separate optimization dynamics. Experiments on ImageNet show improved training stability, codebook utilization, and reconstruction quality.

6 days ago