Against strong decision-theoretic realism

N.B. Some of this post argues by analogy between decision theory and values. I expect at least these parts of the post to be unconvincing to anyone who expects sufficiently smart agents to converge on the same values, as some moral realists do. I will not argue against moral realism here (see e.g. this sequence for one such argument). Note that I take a convergence-based definition of realism for this post. One could hold that there is a truth about the correct decision theory, but that agents won't necessarily converge on it. I don't argue against views like that here. I am interested more in the question of convergence than of truth, because the former bears on whether ASI's decision theory is path dependent. In an upcoming post, I will argue further for path dependence, and for the time sensitivity of interventions to influence AI's decision theory. This is the third post in our sequence Intro to acausal interactions. Introduction In this post, I argue against the following claim, which I call strong decision-theoretic realism: that sufficiently smart agents will all converge on the “correct” decision theory (DT). In doing so, I also argue against a related claim: that absent "lock-in", humans together with somewhat aligned AIs will necessarily converge on a reasonable decision theory. As a consequence of this, I hope to convince the reader that the avoidance of “lock-in” should not be the primary focus when considering decision-theoretic reflection processes.[1] Rather, we should care about the overall quality of the decision-theoretic reflection process (where quality is defined with respect to our object- and meta-level decision-theoretic commitments). As with values, there are certain things that we want to lock in (e.g., fundamental intuitions, some properties regarding how we wish to reflect), and other things that we don’t (e.g., complicated object-level properties that we are relatively uncertain about). There are many possible “reflection processe