The article suggests three strategies to enhance Activation Oracle (AO) performance: providing multiple tokens instead of a single one, sampling multiple times to reduce hallucinations, and using AUC for binary classification tasks. Experiments with Qwen3-8B show that increasing the number of tokens improves AO effectiveness, with 50 tokens surpassing a baseline model.
The chart will appear after repeat observations. The current metric comes from the source.
14 source pointsReal observations only. History before source connection is not reconstructed.
This approach can help improve the reliability and accuracy of activation analysis in language models by addressing common issues like hallucinations and limited context.
The post explores the author's confusion about how neural networks, particularly large language models (LLMs), perform and learn computations. It challenges the conventional 'mechinterp' (mechanistic interpretation) approach that relies on circuit-based explanations, arguing that 'circuits' and 'computation' may not be the most suitable framework for understanding LLMs. The essay discusses representational drift as a key obstacle to weight-based circuit analysis and suggests a 'co-selectionist' view of circuits as emergent units. The author also reflects on how the concept of 'universality' should influence explanations of LLM function.
6 days agoDaily PapersThe paper explores the linearity in Large Language Models (LLMs) by showing that combining inputs from different text streams leads to a superposition of next-token distributions. It suggests that this linearity is an inherent property of the Transformer architecture and can be restored through fine-tuning, allowing for generating two coherent continuations from one forward pass.
4 days agoDaily PapersThe paper introduces Taste-Bench, a benchmark for evaluating an agent's 'taste' in making long-horizon decisions. It measures the ability to choose the better path in decision forks without knowing future outcomes, revealing that current models perform poorly and that taste can be improved through distillation.
6 days agoDaily PapersThis paper introduces FuseReg, a method that replaces heuristic layer fusion in representation autoencoders (RAEs) with training over random subsets of encoder layers. The approach reduces the reconstruction-generation gap by improving robustness to layer fusion choices, achieving higher PSNR and lower generation FID scores without modifying the pretrained encoder.
3 days agoDaily PapersThe study explores JEV-as-a-Judge, a cost-effective method for evaluating language models by making decisions based on confidence levels, showing comparable performance to state-of-the-art judges at a fraction of the cost. It highlights that while JEV performs well on standard tasks, it struggles with complex judgments requiring deeper analysis, but a cascade system that escalates uncertain cases maintains most of the accuracy at lower cost.
6 days agoDaily PapersThis paper introduces StableVQ, a method for improving the training stability of vector-quantized tokenizers by addressing the entanglement between encoder-decoder and codebook training. It proposes three key components: Dynamic STE for encoder stability, Region VQ Loss for codebook learning, and Decoupled Schedule for separate optimization dynamics. Experiments on ImageNet show improved training stability, codebook utilization, and reconstruction quality.
6 days ago