Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value. Fortunately, we haven't been completely stupid, and have given it a rich definition of shareholder value, encompassing the success, flourishing, safety, preference satisfaction, and moral values of the shareholders, not just the stock price. So it won't destroy the company - or the world - to build a giant stock-ticker with a swiftly increasing number on it. This rich definition is contained in a sophisticated LLM, able to analyse the current situation and answer any question about the objective. The LLM is an integral part of the AI's system and motivation, not something it can simply set aside or hack directly. Unfortunately, as discussed, there are going to be zones in the space of possible situations where the LLM's understanding of the goal is going to come apart from ours. The true goal G(true) in those zones is distinct from the LLM's proxy goal G(proxy). And a subset of those "zones of divergence" will be the Goodhart zones - areas where G(proxy) is much easier to satisfy than G(true). The AI will be motivated to steer towards the Goodhart zones, as this lets it satisfy its stated goals extremely well (there are also anti-Goodhart zones, where G(proxy) is much harder to satisfy than G(true); but the AI will have no motivation to steer towards these zones). These Goodhart zones encompass situations where, for instance, the LLM analyst will see the situational data and conclude that destroying the company to build a giant stock ticker is the way to go. Danger areas are safely far away Now, if this were not an LLM but a generic reward-assessing module, then the AI CEO might find it tricky to reach a Goodhart zone. The zone would be distant and unusual, difficult to find, and it could be hard to justify the first steps in that direction. But neural nets and LLMs are, as Eric Drexler put it, "too linear". They are non-linear, of course (or else they wouldn't work) but only just
Translation pending · showing the source descriptionThe chart will appear after repeat observations. The current metric comes from the source.
16 source pointsReal observations only. History before source connection is not reconstructed.
In a post by Nick Cammarata, it is discussed that neural networks may be understandable thanks to modern interpretability tools. The author claims that the majority of neurons in models with trillions of parameters are interpretable, and their circuits are beautiful. He also notes that neurons previously considered complex were misinterpreted.
6 days agoLessWrongThe post critiques the article on social reality in China, highlighting an unbalanced comparison and a lack of systematic analysis. The author emphasizes that Chinese intellectual communities have their own distinct characteristics, different from Western ones, and that values such as knowledge, truth, and technology ethics are also present in China.
5 days agoLessWrongThe author argues that owning large amounts of capital may lead to losing influence earlier than others, through mental breakdown. This is because people seek to manipulate the wealthy to gain resources, making it harder to maintain values and accurate beliefs.
2 days agoHacker NewsAn upcoming critical security update for PHP will address vulnerabilities CVE-2026-109618 and CVE-2026-109619. More information is available on the websites listed in the description.
22 hours agoLobstersAn upcoming security release for the PHP programming language, likely including vulnerability fixes.
20 hours agoHacker NewsMozilla's blog reports on the compromise of the eMudhra intermediate certificate authority. Additional details are available in the bug report.
yesterday