This paper introduces SCOPD, a framework for efficient vision-language models that uses sparse-context on-policy self-distillation to improve performance after token pruning. It shows that useful visual information can remain accessible but be used unreliably, and proposes a method to better utilize pruned visual tokens without additional computation or architecture changes.
The chart will appear after repeat observations. The current metric comes from the source.
1 source votesReal observations only. History before source connection is not reconstructed.
This approach could be useful for optimizing vision-language models by improving the reliability of pruned visual information usage without additional computational costs.
This paper introduces a method to enhance large language models' (LLMs) ability to learn from context by using perturbed public documents. The approach involves rewriting documents, generating reasoning-based questions, and training a student model on these synthetic samples, leading to improved performance on CL-bench and broader transfer to tasks like long-context understanding and reasoning.
2 days agoDaily PapersThe paper introduces PISA, a block-sparse attention mechanism with log-linear complexity, designed to scale language models for long contexts. It uses a pyramid Top-K selection strategy to efficiently identify relevant keys, reducing computational cost compared to traditional methods.
4 days agoDaily PapersThe paper explores the linearity in Large Language Models (LLMs) by showing that combining inputs from different text streams leads to a superposition of next-token distributions. It suggests that this linearity is an inherent property of the Transformer architecture and can be restored through fine-tuning, allowing for generating two coherent continuations from one forward pass.
5 days agoDaily PapersThis paper introduces FuseReg, a method that replaces heuristic layer fusion in representation autoencoders (RAEs) with training over random subsets of encoder layers. The approach reduces the reconstruction-generation gap by improving robustness to layer fusion choices, achieving higher PSNR and lower generation FID scores without modifying the pretrained encoder.
4 days agoDaily PapersThe paper introduces GAGAR, a framework for quality-aware credit redistribution in code agent reinforcement learning (RL). It uses dynamic sampling and an SFT-trained agentic grader to rank test-passing trajectories, adjusting advantages to prioritize higher-quality implementations. The method was evaluated on large-scale industrial code agents with significant parameter counts.
3 days agoDaily PapersThe paper introduces Skill2Env, a framework that generates executable environments from skills by using agent capability demands to create task blueprints with objectives, challenges, and evaluation criteria. It also proposes Iterative Task Hardening to improve task difficulty based on solver execution evidence, leading to improved agent performance across benchmarks.
2 days ago