Modern LLMs Have Tiny GPTs Hidden Inside Them

The study explores whether modern large language models (LLMs) contain small internal models of GPT2 to improve next-token prediction. The experiment involved generating text with GPT2 and then using Qwen models to continue the generated text, testing if Qwen could predict GPT2 completions.