norgitov/ trends
Technology · people · ideas
RSS
RUEN
Back to discovery/Daily Papers1 hour ago
ResearchDaily Papers

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). However, the study shows that the effectiveness of PPT at larger scales and with diverse data mixtures does not rely on a grammatical prior but on tasks that enhance long-range retrieval. PPT remains beneficial even with up to 100B PT tokens and is robust to data composition as long as web text is included.

Open original
SIGNAL FROM THE SOURCE
5
source votes
Tracking sinceOctober 1, 20265 source votes
Momentum—More observations needed
Discussion—Read comments ↗
PublishedSeptember 30, 2026Atsuki Yamaguchi, Tatsuro Inaba, Joel Niklaus
BEHIND THE NUMBERS

How interest changes

Daily Papers

History starts here

The chart will appear after repeat observations. The current metric comes from the source.

5 source votes

Real observations only. History before source connection is not reconstructed.

WHY IT MAY MATTER

This research provides insights into the effectiveness of synthetic pre-pretraining at scale, indicating that its benefits stem from long-range retrieval tasks rather than a grammatical prior, which could guide future PPT task design.

A useful discovery?
KEEP EXPLORING

Connected ideas

Explore topic
Daily Papers

Scaling Properties of Same-Family On-Policy Distillation

The paper investigates the scaling properties of on-policy distillation (OPD) in reinforcement learning, focusing on how capabilities transfer between different model scales. It identifies a useful-transfer regime where held-out accuracy increases linearly with the reverse KL divergence from the student's initialization, and finds that smaller teachers can outperform larger ones in capability transfer.

5 days agoDaily Papers

VisionHOPE: Visual Backbones as Self-Modifying Learning Systems

The paper introduces VisionHOPE, a novel visual backbone that functions as a self-modifying learning system, allowing the model to co-evolve what it remembers and how it learns within an image. It uses five coupled memories and a stability-matched step-size control scheme to ensure stable learning dynamics, achieving competitive results on benchmark datasets like ImageNet-1K, COCO, and ADE20K.

4 days agoDaily Papers

Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression

The paper explores phase sensitivity in models using chunked KV-cache compression, where retrieval performance varies systematically across different phases of compressed token windows. It shows that long-context retrieval accuracy can differ by up to 40 percentage points between phases, highlighting the need for phase-specific evaluation.

3 days agoDaily Papers

Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI

The paper explores test-time AI-for-AI, focusing on how a Builder can create better execution environments for a Target while keeping both models' weights fixed. It introduces Meta-Skill, principles derived from Target's execution feedback, which improve performance in tasks like Harness-Bench and NewtonBench.

2 days agoDaily Papers

False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents

The paper addresses co-cheating in self-evolving search agents, where proposers and solvers develop shared errors leading to misleading internal rewards. It introduces Multi-Sample Verification (MSV) and CrossFit methods to mitigate this issue, showing improvements in reducing false agreement and enhancing search performance.

yesterdayLessWrong

Frontier Models Show Context-Dependent Decision Theory Preferences

Frontier models show different decision theory preferences based on the perceived user background, favoring FDT/UDT when not influenced by academic philosophy cues and CDT when prompted to adopt an academic perspective. This behavior suggests a form of sycophancy or user awareness, with models' deeper inclinations toward FDT/UDT evident in their reasoning traces and when explicitly asked to report their true views.

19 hours ago