Against Moloch #48

How much do you trust your model—would you even know if it was misbehaving? For that matter, how much do you trust your frontier lab? We desperately need better visibility into every part of the AI stack. This week brings us cautious good news: perhaps neural networks are easier to understand than we’d previously believed. And Anthropic is moving forward with embedded independent evaluators, with reason to hope that at least some of the other leading labs will follow suit. Top pick Neural networks are already clean The big news this week is Nick Cammarata’s essay Neural networks are already clean. Nick did some of the seminal work on polysemanticity and is one of the people responsible for the current widespread belief that neuron-level interpretability is almost impossibly difficult. He now argues that he was mistaken: with the right tools and techniques, most of the neurons in modern models can be understood: > I think I have good news about the nature of neural networks. > After many months studying neurons and circuits in image and language models, I’ve come to believe a network’s own mechanisms are its legible building blocks, and that the outlook for mechanistic interpretability may be far better than I’d feared. The essay itself is massive and technical. If that’s more than you signed up for, I recommend Zvi’s report, or MIT Technology Review’s interview with Nick. How important is this work? Opinions vary wildly. roon (OpenAI): > when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even eliezer will have to lower his pdoom. we can start making alignment into a 'normal science' Neel Nanda (Google DeepMind): > This is interesting work, but I don't think it's a significant update to the big picture of interpretability. My take is that we just don’t know yet. This is strong and important work that gives us new hope for mech interp,