Preface to the Computational Agent Theory and ASI Alignment Sequence

The idea of “artificial superintelligence [1] ” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the field of AI. Unfortunately, ASI poses a grave existential risk to the human species. This was argued by Eliezer Yudkowsky since 2001 [2] , and by many others (see e.g., Omohundro 2009, Bostrom 2014, Armstrong 2014, Russell 2019, Bengio et al. 2024 (25 authors), Bengio et al. 2025 (13 authors) and Yudkowsky and Soares 2025). In light of this risk comes the question of how can we make ASI safe [3] . This question is challenging on many levels, because to solve it we need to complete vast gaps in our understanding both in theoretical computer science and in the areas of philosophy concerned with knowledge and decision-making. Moreover, the cost of failure is so enormous that any sane solution has to meet unprecedented standards of reliability and robustness. Not to mention the sociopolitical problem of balance of power in the post-ASI age, something that is out-of-scope for this sequence. Personally, I started to be concerned with this problem around 2013. In 2018, I created the learning-theoretic AI alignment research agenda (LTA). The LTA combined ideas from MIRI’s agent foundations agenda with the mathematical tools of statistical and computational learning theory in an attempt to create a mathematical theory of intelligent agents. Since then, my collaborators [4] and I have been making steady progress on realizing this vision. In 2023, I wrote an updated overview of the agenda, including a sketch of my proposed alignment protocol: Computational Superimitation. More recently I founded the Computational Rational Agents Laboratory (CORAL), a dedicated research organization [5] . This year has seen a breakthrough in a key missing piece of LTA: compositional learning theory (see Stringological Sequence Prediction, more on