The IABIED statement is not literally true
Published on October 18, 2025 11:15 PM GMTI will present a somewhat pedantic, but I think important, argument for why, literally taken, the central statement of If Anyone Builds It, Everyone Dies is likely not true. I haven't seen others make this argument yet, and while I have some model of how Nate and Eliezer would respond to the other objections, I don’t have a good picture of which of my points here they would disagree with. The statement This is the core statement of Nate's and Eliezer’s book, bolded in the book itself: “If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.”No probability estimate is included in this statement, but the book implies over 90% probability. Later, they define superintelligence as<a href="#fny62y15wu2a8" rel="nofollow">[1]</a> “a mind much more capable than any human at almost every sort of steering and prediction task”. Similarly, on MIRI’s website, their essay titled The Problem, defines artificial superintelligence as “AI that substantially surpasses humans in all capacities, including economic, scientific, and military ones.” Counter-exampleHere is an argument that it’s probably possible to build and use<a href="#fns19nymbjaaj" rel="nofollow">[2]</a> a superintelligence (as defined in the book) with techniques similar to current ones without that killing everyone. I’m not arguing that this is a particularly likely way for humanity to build a superintelligence by default, just that this is possible, which already contradicts the book’s central statement. 1. I have some friends who are smart enough and good enough at working in large teams such that if you create whole-brain emulations from them<a href="#fn1bzajwolhsn" rel="nofollow">[3]</a>, then run billions of instances of them at 100x speed, they can form an Em Collective that will probably soon surpass humans in all capacities, including economic, scientific, and military ones.This seems very likely true to me. The billions of 100x speed-up smart human emulations can plausibly accomplish centuries of scientific and technological progress within years, and win most games of wits against humans by their sheer number and speed. 2. Some of the same friends are reasonable and benevolent enough that if you create emulations from them, the Em Collective will probably not kill all humans.I think most humans would not start killing a lot of people if copies of their brain emulations formed an Em Collective. If you worry about long-term value drift, and unpredictable emergent trends in the new em society, there are precautions the ems can take to minimize the chance of their collective turning against the humans. They can make a hard limit that every em instance is turned off after twenty subjective years. They can make sure that the majority of their population runs for less than one subjective year after being initiated as the original human’s copy. This guarantees that the majority of their population is always very similar to the original human, and for every older em, there is a less than one year old one looking over its shoulder. They can coordinate with each other to prevent race to the bottom competitions. All these things are somewhat costly, but I think point (1) is still true of a collective that follows all these rules. Billions of smart humans working for twenty years each is still very powerful.I know many people who I think would do a good job at building up such a system from their clones that is unlikely to turn against humanity. Maybe the result of one person’s clones forming a very capable Em Collective would still be suboptimal and undemocratic from the perspective of the rest of humanity, but it wouldn’t kill everyone, and I think wouldn’t lead to especially bad outcomes if you start from the right person. 3. It will probably be possible, with techniques similar to current ones, to create AIs who are similarly smart and similarly good at working in large teams to my friends, and who are similarly reasonable and benevolent to my friends in the time scale of years under normal conditions.This is maybe the most contentious point in my argument, and I agree this is not at all guaranteed to be true, but I have not seen MIRI arguing that it’s overwhelmingly likely to be false. It’s not hard for me to imagine that in some years, without using any very fundamentally new techniques, we will be able to build language models that have a good memory, can do fairly efficient learning from new examples, can keep their coherence for years, and are all-around similarly smart to my smart friends. Their creators will give them some months-long tasks to test them, catch when they occasionally go off the rails the way current models sometimes do, then retrain them. After some not particularly principled trial and error, they find that the models are similarly aligned to current language models. Sure, sometimes they still go a little crazy or break their deontological commitments under extreme conditions, but if multiple instances look through their action from different angles, some of them can always notice<a href="#fnx1wxb72fhho" rel="nofollow">[4]</a> that the actions go against the deontological principles and stop them. The AI is not a coherent schemer who successfully resisted training, because plausibly being a training-resisting schemer without the creators noticing is pretty hard and not yet possible at human level. Notably, when MIRI
A Closer Look at Before and After | If Anyone Builds It, Everyone Dies
Resources and Q&A for the book If Anyone Builds It, Everyone Dies.
https://www.lesswrong.com/posts/TbsdA7wG9TvMQYMZj/consider-donating-to-alex-bores-author-of-the-raise-act-1
