Pretraining Data, Not Verifiability, Explains LLMs' Strength in Math and Coding

The article argues that large language models (LLMs) excel at math and coding not because these tasks are easy to verify, but due to the high quality of pretraining data. It suggests that the accuracy of math literature and functional code on the internet allows LLMs to imitate correct patterns effectively. However, the article notes that while LLMs can generate code that works, it may still be inefficient or buggy, requiring further refinement.