Game theory of "Nuclear Prisoner's Dilemma" - on nuking rocks Published on May 12, 2025 11:07 AM GMTEliezer Yudkowsky wrote, in a place where I can't ask a follow-up question: A rational agent should always do at least as well for itself as a rock, unless it's up against some other agent that specifically wants to punish particular decision algorithms and will pay costs itself to do that; just doing what a rock does isn't very expensive or complicated, so a rational agent which isn't doing better than a rock should just behave like a rock instead. An agent benefits from building into itself a capacity to respond to positive-sum trade offers; it doesn't benefit from building into itself a capacity to respond to threats. Consider the Nuclear Prisoner's Dilemma, in which as well as Cooperate and Defect there's a third option called Nuke, which if either player presses it causes both players to get (-100, -100). Suppose that both players are programs each allowed to look at each other's source code (a la our paper "Robust Cooperation in the Prisoner's Dilemma"), or political players with track records of doing what they say. If you're up against a naive counterparty, you can threaten to press Nuke unless the opponent presses Cooperate (in which case you press Defect). But you'd have no reason to ever press Nuke if you were facing a rock; the only reason you'd ever set up a strategy of conditionally pressing Nuke is because of a prediction about how your opponent would respond in a complicated way to that strategy by their pressing Cooperate (even though you would then press Defect, and they'd know that). So a rational agent does not want to build into itself the capacity to respond to threats of Nuke by choosing Cooperate (against Defect); it would rather be a rock. It does want to build into itself a capacity to move from Defect-Defect to Cooperate-Cooperate, if both programs know the other's code, or two entities with track records can negotiate. Well, what if I told you that I had a perfectly good reason to to become someone that would threaten to nuke Defection Rock, and that it was because I wanted to make it clear that agents that self-modify into a rock get nuked anyway, so there's no advantage to adopting a strategy that does something other than playing Cooperate while I play Defect. I want to keep my other victims convinced that surrendering to me is their best option, and nuking the occasional rock is a price I'm willing to pay to achieve that. In other words, I've transformed the game you're playing from Prisoner's Dilemma to Hawk/Dove, and I'm a rock that always plays Hawk. So what does LDT have to say about that? Are you going to use a strategy that plays "Hawk" (anything other than Cooperate) against a rock that always plays Hawk and gets us both nuked, or are you going to do the sensible thing and play Dove (Cooperate)? https://www.lesswrong.com/posts/Pu9YPzeuJGxyKpcJZ/untitled-draft-74o4#comments https://www.lesswrong.com/posts/Pu9YPzeuJGxyKpcJZ/untitled-draft-74o4
What Is Death? Published on May 12, 2025 2:14 AM GMTThis article is an excerpt from the book 'The Future Loves You'.Summary of the article by https://www.lesswrong.com/posts/dqmW7CjdsoDLxXwPd/what-is-death
Highly Opinionated Advice on How to Write ML Papers Published on May 12, 2025 1:59 AM GMTTL;DRThe essence of an ideal paper is the narrative: a short, rigorous and evidence-based technical story you tell, with a takeaway the readers care aboutWhat? A narrative is fundamentally about a contribution to our body of knowledge: one to three specific novel claims that fit within a cohesive themeHow? You need rigorous empirical evidence that convincingly supports your claimsSo what? Why should the reader care?What is the motivation, the problem you’re trying to solve, the way it all fits in the bigger picture?What is the impact? Why does your takeaway matter? The north star of a paper is ensuring the reader understands and remembers the narrative, and believes that the paper’s evidence supports itThe first step is to compress your research into these claims.The paper must clearly motivate these claims, explain them on an intuitive and technical level, and contextualise what’s novel in terms of the prior literatureThis is the role of the abstract & introductionExperimental Evidence: This is absolutely crucial to get right and aggressively red-team, it’s how you resist the temptation of elegant but false narratives.Quality > Quantity: find compelling experiments, not a ton of vaguely relevant ones.The experiments and results must be explained in full technical detail - start high-level in the intro/abstract, show results in figures, and get increasingly detailed in the main body and appendix.Ensure researchers can check your work - provide sufficient detail to be replicatedDefine key terms and techniques - readers have less context than you think.Write iteratively: Write abstract -> bullet point outline -> introduction -> first full draft -> repeatGet feedback and reflect after each stageSpend comparable amounts of time on each of: the abstract, the intro, the figures, and everything else - they have about the same number_of_readers * time_to_readInform, not persuade: Avoid the trap of overclaiming or ignoring limitations. Scientific integrity may get you less hype, but gains respect from the researchers who matter.Precision, not obfuscation: Use jargon where needed to precisely state your point, but not for the sake of sounding smart. Use simple language wherever possible.<img src="https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/081539a4d3b61418cfc6657e29bac2d234e1adf5331b4a7b3ea7974ad4d2bfc1/pact399wy3qd43yhrzx8" alt="">Case study: The abstract of ? https://www.lesswrong.com/posts/eJGptPbbFPZGLpjsp/highly-opinionated-advice-on-how-to-write-ml-papers
Consider not donating under $100 to political candidates Published on May 11, 2025 3:20 AM GMTEpistemic status: thing people have told me that seems right. Also primarily relevant to US audiences. Also I am speaking in my personal capacity and not representing any employer, present or past. Sometimes, I talk to people who work in the AI governance space. One thing that multiple people have told me, which I found surprising, is that there is apparently a real problem where people accidentally rule themselves out of AI policy positions by making political donations of small amounts—in particular, under $10. My understanding is that in the United States, donations to political candidates are a matter of public record, and that if you donate to candidates of one party, this might look bad if you want to gain a government position when another party is in charge. Therefore, donating approximately $3 can significantly damage your career, while not helping your preferred candidate all that much. Furthermore, at the time you make this donation, you might not realize that you will later want to get a government position. Now, I don’t want to overly discourage this sort of thing. It’s your money, free speech is great, and fundamentally I think it’s fine to have and publicly express political views (for example, I think Donald Trump is extremely bad, and am disappointed in my fellow countrymen for voting for him). That said, I think that one should be aware of the consequences of making political donations, and it seems plausible to me that if you’re not willing to donate more than $100 to a political candidate, consider that the career cost to you of making that donation may be higher than the benefit that it confers.https://www.lesswrong.com/posts/tz43dmLAchxcqnDRA/consider-not-donating-under-usd100-to-political-candidates#comments https://www.lesswrong.com/posts/tz43dmLAchxcqnDRA/consider-not-donating-under-usd100-to-political-candidates
Somerville Porchfest 2025 Published on May 11, 2025 2:00 AM GMT <a href="https://somervilleartscouncil.org/porchfest/" rel="nofollow">Somerville Porchfest</a> was a lot of fun this year, and we got a great crowd: image Showing some pictures to Claude it guesses there might have been 600 people. Very glad the city gave us permission to close the street! [1] It would have been good not having cars parked in the main dance area, since this ended up making us need to spread out a lot more than would be ideal. Next time I'm going to see if I can get friends to park in the four main spots in the ~day leading up to the dance and then move their cars just as we're starting. did a great job leading the dancing, calling a mix of simple dances for everyone and standard contras for experienced dancers. This is also how he did it last year, and I think this does a good job covering the somewhat incompatible things people are looking for out of the event. For the standard contras we had one ~130ft line, while for the simple dances we had two lines of similar length. image I played with Rohan and Charlie, with Weiwei and Rick sitting in. I think we sounded pretty good; a bit of a pickup band, but still fun! Sam sat in for one dance on piano and Weiwei and I got to play trumpet and baritone together. At the end I convinced Sam to try playing the kick drum while I played hi-hat—part of my long-term plan to hook more people on <a href="https://www.jefftk.com/p/introduction-to-heel-toe-drumming" rel="nofollow">foot drumming</a>. Sound was a bit chaotic. I was running sound while also playing, and this is never ideal. I got a few people (thanks Rick and Cecily) to give me feedback on the mix, but next year we should line up a dedicated volunteer for sound. Speaker placement also matters a lot: we started with the speakers on the porch because it hadn't quite stopped raining, but it wasn't possible to get them anywhere near loud enough for the crowd without feedback. As soon as the rain let us we moved them well in front of the band, which let us run them a lot louder without feedback and give better coverage over the crowd. But it was awkwardly loud right in front of the speakers, even though they were ~10ft up. Next time I'd like to try fills: a pair of K10s in the center, and then a pair of K10.2s (with built-in delays) ~30ft up and down the street to help even out the sound. With so many people it would also be helpful to have additional volunteers on crowd control, beyond just Harris at the mic: sometimes it's clearer to show than tell. Another thing to think about for next time! This was the first year where this was officially a event. This mostly was a publicity thing: BIDA announced the dance in advance and then we advertised BIDA during the dance. Possibly it should keep being a BIDA event? I'll need to see what the board thinks. I don't envy the situation the organizers were in with rain dates: delaying it to Mother's Day, which was also a day with a shutting down major streets, wouldn't have been good. But it was also not great that the 12-2 slot was pretty rained out. Perhaps next year ensuring that the rain date is a day without major conflicts could help? Though given the size of the crowds we saw with todays somewhat wet weather, if 2026 is dry it might be really quite a large festival. My impression is that the new system where a grid of major streets were off-limits for playing helped a lot with allowing people to move around the city if they needed to? Looking forward to 2026! [1] I messed this up and almost didn't get permission: I applied to play (first year with applications) and was thinking that if we did get in then I'd apply for a block party permit. What I didn't realize was that for 2025 they were also changing things so the deadline for Porchfest block party permits was the same day as the application deadline! I applied late and was initially denied, but then I wrote to SAC to explain the situation and they very nicely allowed my late application to go through. https://www.lesswrong.com/posts/9rurcdtdvEYqpYbKs/somerville-porchfest-2025#comments https://www.lesswrong.com/posts/9rurcdtdvEYqpYbKs/somerville-porchfest-2025
Book Review: "Encounters with Einstein" by Heisenberg Published on May 10, 2025 8:55 PM GMTThe title of this book is clickbait. "Encounters with Einstein" is a short collection of lectures given or written by Werner Heisenberg in the 1970's, only one of which discusses Einstein.<a href="#fnuw3t6vb218" rel="nofollow">[1]</a> The remaining lectures discuss various aspects of the history and development of quantum mechanics and science in general: tradition and concepts in science; Heisenberg's time in Göttingen, where he, Born and Jordan developed matrix mechanics; a history of cosmic ray science, and more. Concepts and QuarksOne thread comes up repeatedly in the lectures: the role of concepts in science, and in particular, the concept of an elementary particle. (You get the sense that he would harp on this point to whoever would listen.) Much progress in science has come from asking what things consist of, and whether they can be broken down into smaller pieces: molecules, atoms, then electrons and protons in the early 20th century. It was believed that these elementary particles were truly immutable, that their number was always constant. This proved incorrect, though. The discoveries of matter-antimatter pair production and nuclear decay showed that while mass-energy, momentum, charge, etc. might be conserved, the number of particles was not. Cosmic ray events, in which the collision of two highly energetic particles releases a shower of secondary particles through the transformation of energy into matter, also demonstrated this conclusively. Heisenberg argues that it thus makes little sense to talk of "dividing" particles into their constituent parts any longer:A proton, for example, could be made up of neutron and pion, or Λ-hyperon and kaon, or out of two nucleons and an anti-nucleon; it would be simplest of all to say that a proton just consists of continuous matter, and all these statements are equally correct or equally false. The difference between elementary and composite particles has thus basically disappeared. (p. 73)The pervasive Democritean idea that everything can be decomposed into some indivisible, immutable "atoms" must, he claims, be rejected: "good physics is unconsciously being spoiled by bad philosophy." How should we instead conceptualize matter? He prefers to describe it as a "spectrum," analogous to the spectra of atoms or molecules. The different particles that we observe correspond to particular stationary states, with transformations between them determined by the relevant symmetries and conservation laws. These symmetries and the dynamics they lead to would then be the primary objects of study for physics.This leads us to the theory of quarks.<a href="#fneuj5sa1l8gj" rel="nofollow">[2]</a> Heisenberg is not a fan:Here the question has obviously been asked, "What do protons consist of?" But it has been forgotten in the process, that the term consist of only has a halfway clear meaning if we are able to dissect the particle in question, with a small expenditure of energy, into constituents whose rest mass is very much greater than this energy-cost; otherwise, the term consist of has lost its meaning. And that is the situation with protons. (p. 83)He addresses the experimental evidence in favor of quarks,<a href="#fn1uki0lbs59kj" rel="nofollow">[3]</a> arguing that just because a theory can explain some narrow set of experiments does not mean we should immediately jump to adopt it. He draws an analogy to the pre-Bohr theory that atoms are composed of harmonic oscillators. Woldemar Voigt used this theory to develop complex formulas predicting certain properties of the optical spectra of atoms, predictions which closely matched experiments. When Heisenberg and Jordan later tried to model these properties using quantum mechanics, they found the exact same formulas. "But," he says, "the Voigtian theory contributed nothing to the understanding of atomic structure." The equivalence between the oscillator theory and quantum mechanics was a purely formal, mathematical one, as they both lead to a system of coupled linear equations whose parameters Voigt could pick to match experiments.Voigt had made phenomenological use of a certain aspect of the oscillator hypothesis, and had either ignored all the other discrepancies of this model, or deliberately left them in obscurity. Thus he had simply not taken his hypothesis in real earnest. In the same way, I fear that the quark hypothesis is just not taken seriously by its exponents. The questions about the statistics of quarks, about the forces that hold them together, about the particles corresponding to these forces, about the reasons why quarks never appear as free particles, about the pair-creation of quarks in the interior of the elementary particle -- all these questions are more or less left in obscurity. (p. 85)I don't know nearly enough physics to evaluate Heisenberg's arguments on a technical level (I'd be interested to hear from people who do!). Conceptually, though, they seem fairly convincing. Of course, reality turned out the other way: quarks are by now more or less universally accepted as part of the Standard Model of particle physics. Heisenberg died in 1976; I have to wonder at what point he would have updated had he lived to see the theory fully developed.Eureka MomentsThe book covers a range of other topics; most interesting to me was Heisenberg's discussion of the development of new theories in science. He asks, "why, at the very first moment of their appearance, and especially for him who first sees them, the correct closed theory possesses an enormous persuasive power, long before the conceptual or even the mathematical foundations are completely clarified, and long before it could be said that many experiments had confirmed it?" (By "correct closed theory" he means a consistent mathematical theory that fully and accurately describes some large set of phenomena, at least within some bounded conceptual domain: he gives Newtonian mechanics, special relativity, Gibbs's theory of heat, and the Dirac-von Neumann axiomatization of quantum mechanics as examples.) Heisenberg's somewhat cryptic answer is that "the conceptual systems under consideration undoubtedly form a discrete, not a continuous, manifold." In other words:In all probability, the decisive precondition here is that the physicists most intimately acquainted with the relevant field of experience have felt very clearly, on the one hand, that the phenomena of this field are closely connected and cannot be understood independently of each other, while this very connection, on the other, resists interpretation within the framework of existing concepts. The attempt, nonetheless, to effect such an interpretation, has repeatedly led these physicists to assumptions that harbor contradictions, or to wholly obscure distinctions of cases, or to an impenetrable tangle of semi-empirical formulae, of which it is really quite evident that they cannot be correct... The surprise produced by the right proposal, the discovery that "yes, that could actually be true," thus gives it from the very outset a great power of persuasion. (p. 127-129)This sounds very much like a description of Thomas Kuhn's crises and paradigm shifts, translated into a much more personal and psychological setting. "The Structure of Scientific Revolutions" was published in 1962; Heisenberg doesn't cite it<a href="#fnhv5wrtg89wu" rel="nofollow">[4]</a> but he could very well have been familiar with its ideas.Heisenberg doesn't mention quarks in this chapter, but they would seem to be a good example of what he describes: a set of confusing but highly interrelated observations and a theory that unifies and explains many of them. (Quarks were initially proposed as a way of simplifying and organizing the extensive "particle zoo" of putative elementary particles that physicists had identified by the 60's.<a href="#fn4kg72ohekai" rel="nofollow">[5]</a>) I don't know what Gell-Mann and Zweig experienced when they proposed the quark model in 1964, whether they had the sort of "eureka" moment that Heisenberg describes. (Obviously, the theory did not have any sort of "enormous persuasive power" for Heisenberg.) Of course, just like many other scientific revolutions, the adoption of quark theory was a slow, messy process, regardless of how its originators may have felt. The original proposals only included three flavors of quark (up, down, strange): the other three (charm, top, bottom) were added to the theory over the subsequent decade. And as Heisenberg points out, the theory had some major theoretical and empirical weaknesses at first, which were only slowly resolved. Regardless, the chapter is an interesting discussion of the psychological process of scientific discovery from someone with extensive first-hand experience.Oh yeah, EinsteinThe lecture from which the book takes its overall title opens with the disclaimer that "The word encounters here must be taken to refer, not only to personal meetings, but also to encounters with Einstein's work." (I suppose with that definition the title can indeed be applied to the entire book, as well as to most of 20th century physics.) He did meet Einstein a few times, though, each time struggling to convince him of the conceptual shifts required by quantum theory. In 1926 he discussed with Einstein how quantum mechanics contains no notion of an "electron path," with electrons instead jumping discontinuously between states. In 1927 at the famous Solvay Conference he, Bohr, and Pauli contended with Einstein's attacks on the uncertainty principle. And much later in 1954 he argued with him about whether quantum theory must be a consequence of some unified field theory, as Einstein thought, or whether quantum theory was itself fundamental. Einstein, of course, famously stuck to his maxim that "God does not play dice" for the rest of his life.Negative SpaceOne thing very prominently not discussed in the book is what Heisenberg did during the Second World War. Unlike most top scientists, he stayed in Germany throughout the war. He alludes to this only once, mentioning as an aside that his institute "was engaged during the war in work on the construction of an atomic reactor." That's all we get, though, so this story will have to wait for a different book review.<a href="#fnr88b3c7p2uf" rel="nofollow">[6]</a>ConclusionHeisenberg is right, of course, about the importance of conceptual frames in science and in human reasoning in general. Progress often comes about by the introduction of new concepts or new ways of thinking about a domain. And both Einstein and, unknowingly, Heisenberg himself demonstrate the risk of becoming too attached to a set of concepts that don't match reality. How did Einstein and Heisenberg go so wrong? Heisenberg, at least, might have the excuse of age: he was in his late 60's and 70's when the theory of quarks was developed. But Einstein spent decades struggling with quantum physics.I don't have a great answer to this question. I'm sure an immense amount has been written about Einstein's failure to accept quantum physics, of which I've read very little. For me, the important point is whether Einstein and Heisenberg changed -- maybe their research taste degraded with age, or they became too invested in particular theories, or too confident in their own instincts -- or whether they simply stayed the same while physics moved past them. If it's the former, then the lesson to take away from this is one of intellectual humility (and also that we should cure aging). If it's the latter, though, then their example suggests to me that research taste may be much more contextual than we typically think.<a href="#fn5nfdi9bka3l" rel="nofollow">[7]</a> Maybe they developed some set of mental tools, implicit or explicit conceptual frames, etc. that were very well-suited to the scientific problems they dealt with in their youth, but which failed them when physics moved on to new settings. It's tempting to take away the lesson that one should try to explore widely and not become too attached to any one way of doing things. But this is the classic https://www.lesswrong.com/posts/hc7qi5u2XszzrXaPf/book-review-encounters-with-einstein-by-heisenberg
Where is the YIMBY movement for healthcare? Published on May 10, 2025 8:36 PM GMTIn the progress movement, some cause areas are about technical breakthroughs, such as fusion power or a cure for aging. In other areas, the problems are not technical, but social. Housing, for instance, is technologically a solved problem. We know how to build houses, but housing is blocked by law and activism.The YIMBY movement is now well established and gaining momentum in the fight against the regulations and culture that hold back housing. More broadly, similar forces hold back building all kinds of things, including power lines, transit, and other infrastructure. The same spirit that animates YIMBY, and some of the same community of writers and activists, has also been pushing to reform regulation such as NEPA.Healthcare has both types of problems. We need breakthroughs in science and technology to beat cancer, heart disease, neurodegenerative diseases, and aging. But also, healthcare (in the US at least) is far more expensive and less effective than it should be.I am no expert, but I am struck that:The doctor-patient relationship has been disintermediated by not one but two parties: insurers and employers.It is not a fee-for-service relationship. The price system in medicine has been mangled beyond recognition. Patients are not told prices; doctors avoid, even disdain, any discussion of prices; and the prices make no rational sense even if and when you do discover them. This destroys all ability to make rational economic choices about healthcare.Patients often switch insurers, meaning that no insurer has an interest in the patient's long-term health. This is a disaster in a world where most health issues build up slowly over decades and many of them are affected by lifestyle choices.Insurers are highly regulated in what types of plans they can offer and in what they can and cannot cover. There's no real room for insurer creativity or consumer choice, or for either party to exercise judgment.A lot of money is spent at end of life, with little gained by in many cases except a few years or months (if that) of a painful, bedridden existence.Just to name a few. https://www.lesswrong.com/posts/Ha2EFzxpFLCec3vTs/where-is-the-yimby-movement-for-healthcare
Jim Babcock's Mainline Doom Scenario: Human-Level AI Can't Control Its Successor Published on May 9, 2025 5:20 AM GMTEliezer's AI doom arguments have had me convinced since the ancient days of 2007, back when AGI felt like it was many decades away, and we didn't have an intelligence scaling law (except to the Kurzweilians who considered Moore's Law to be that, and were, in retrospect, arguably correct).Back then, if you'd have asked me to play out a scenario where AI passes a reasonable interpretation of the Turing test, I'd have said there'd probably be less than a year to recursive-self-improvement FOOM and then game over for human values and human future-steering control. But I'd have been wrong.Now that reality has let us survive a few years into the "useful highly-general Turing-Test-passing AI" era, I want to be clear and explicit about how I've updated my mainline AI doom scenario.So I interviewed Jim Babcock (https://www.lesswrong.com/users/jimrandomh?mention=user https://www.lesswrong.com/posts/nZtN9PW4qBeNKehfs/jim-babcock-s-mainline-doom-scenario-human-level-ai-can-t