In recent decades, a growing coalition has emerged to oppose the development of artificial intelligence technology, for fear that the imminent development of smarter-than-human machines could doom humanity to extinction. The now-influential form of these ideas began as debates among academics and internet denizens, which eventually took form—especially within the Rationalist and Effective Altruist movements—and grew in intellectual influence over time, along the way collecting legible endorsements from authoritative scientists like Stephen Hawking and Geoffrey Hinton.

Ironically, by spreading the belief that superintelligent AI is achievable and supremely powerful, these “AI Doomers,” as they came to be called, inspired the creation of OpenAI and other leading artificial intelligence labs whose technology they argue will destroy us all. Despite this, they have continued nearly the same advocacy strategy, and are now in the process of persuading Western governments that superintelligent AI is achievable and supremely powerful. To this end, they have created organized and well-funded movements to lobby for regulation, and their members are staffing key positions in the U.S. and British governments.

Their basic argument is that more intelligent beings can outcompete less intelligent beings, just as humans outcompeted mastodons or saber-toothed tigers or neanderthals. Computers are already ahead of humans in some narrow areas, and we are on track to create a superintelligent artificial general intelligence (AGI) which can think as broadly and creatively in any domain as the smartest humans. “Artificial general intelligence” is not a technical term, and is used differently by different groups to mean everything from “an effectively omniscient computer which can act independently, invent unthinkably powerful new technologies, and outwit the combined brainpower of humanity” to “software which can substitute for most white-collar workers” to “chatbots which usually don’t hallucinate.” 

AI Doomers are concerned with the former scenario, where computer systems outreason, outcompete, and doom humanity to extinction. The AI Doomers are only one of several factions that oppose AI and seek to cripple it via weaponized regulation. There are also factions concerned about “misinformation” and “algorithmic bias,” which in practice means they think chatbots must be censored to prevent them from saying anything politically inconvenient. Hollywood unions oppose generative AI for the same reason that the longshoremen’s union opposes automating American ports and insists on requiring as much inefficient human labor as possible. Many moralists seek to limit “AI slop” for the same reasons that moralists opposed previous new media like video games, television, comic books, and novels—and I can at least empathize with this last group’s motives, as I wasted much of my teenage years reading indistinguishable novels in exactly the way that 19th century moralists warned against. In any case, the AI Doomers vary in their attitudes towards these factions. Some AI Doomers denounce them as Luddites, some favor alliances of convenience, and many stand in between.

Most members of the “AI Doomer” coalition initially called themselves by the name of “AI safety” advocates. However, this name was soon co-opted by these other factions with concerns smaller than human extinction. The AI Doomer coalition has far more intellectual authority than AI’s other opponents, with the most sophisticated arguments and endorsements from socially-recognized scientific and intellectual elites, so these other coalitions continually try to appropriate and wield the intellectual authority gathered by the AI Doomer coalition. Rather than risk being misunderstood, or fighting a public battle over the name, the AI Doomer coalition abandoned the name “AI safety” and rebranded itself to “AI alignment.” Once again, this name was co-opted by outsiders and abandoned by its original membership. Eliezer Yudkowsky coined the term “AI Notkilleveryoneism” in an attempt to establish a name that could not be co-opted, but unsurprisingly it failed to catch on among those it was intended to describe.

Today, the coalition’s members do not agree on any name for themselves. “AI Doomers,” the only widely understood name for them, was coined by their rhetorical opponents and is considered somewhat offensive by many of those it refers to, although some have adopted it themselves for lack of a better alternative. While I regret being rude, this essay will refer to them as “AI Doomers” in the absence of any other clear, short name.

Whatever name they go by, the AI Doomers believe the day computers take over is not far off, perhaps as soon as three to five years from now, and probably not longer than a few decades. When it happens, the superintelligence will achieve whatever goals have been programmed into it. If those goals are aligned exactly to human values, then it can build a flourishing world beyond our most optimistic hopes. But such goal alignment does not happen by default, and will be extremely difficult to achieve, if its creators even bother to try. If the computer’s goals are unaligned, as is far more likely, then it will eliminate humanity in the course of remaking the world as its programming demands. This is a rough sketch, and the argument is described more fully in works like Eliezer Yudkowsky’s essays and Nick Bostrom’s Superintelligence.

This argument relies on several premises: that superintelligent artificial general intelligence is philosophically possible, and practical to build; that a superintelligence would be more or less all-powerful from a mere human perspective; that superintelligence would be “unfriendly” to humanity by default; that superintelligence can be “aligned” to human values by a very difficult engineering program; that superintelligence can be built by current research and development methods; and that recent chatbot-style AI technologies are a major step forward on the path to superintelligence. Whether those premises are true has been debated extensively, and I don’t have anything useful to add to that discussion which I haven’t said before. My own opinion is that these various premises range from “pretty likely but not proven” to “very unlikely but not disproven.”

Even assuming all of this, the political strategy of the AI Doomer coalition is hopelessly confused and cannot possibly work. They seek to establish onerous regulations on for-profit AI companies in order to slow down AI research—or forcibly halt research entirely, euphemized as “Pause AI,” although most of the coalition sees the latter policy as desirable but impractical to achieve. They imagine that slowing or halting development will necessarily lead to “prioritizing a lot of care over moving at maximal speed” and wiser decisions about technology being made. This is false, and frankly very silly, and it’s always left extremely vague because the proponents of this view cannot articulate any mechanism or reason why going slower would result in more “care” and better decisions, with the sole exception of Yudkowsky’s plan to wait indefinitely for unrelated breakthroughs in human intelligence enhancement.

But more immediately than that, if AI Doomer lobbyists and activists like the Center for AI Safety, the Institute for AI Policy and Strategy, Americans for Responsible Innovation, Palisade Research, the Safe AI Forum, Pause AI, and many similar organizations succeed in convincing the U.S. government that AI is the key to the future of all humanity and is too dangerous to be left to private companies, the U.S. government will not simply regulate AI to a halt. Instead, the U.S. government will do what it has done every time it’s been convinced of the importance of a powerful new technology in the past hundred years: it will drive research and development for military purposes. This is the same mistake the AI Doomers made a decade ago, when they convinced software entrepreneurs that AI is the key to the future and so inspired them to make the greatest breakthroughs in AI of my lifetime. The AI Doomers make these mistakes because their worldview includes many assumptions, sometimes articulated and sometimes tacit, which don’t hold up to scrutiny.

There is No Reason to Think Superintelligence is Coming Soon

AI Doomers believe that superintelligent AGI is coming very soon, generally within five or ten years. They engage each other in a grand debate over “timelines” to such an AGI, despite the lack of any solid grounding for this debate. They rarely advance concrete arguments for any particular timelines, and when they do, the arguments are not sound. More often, they justify this position by pointing to the loose vibes and intuitions of technologists and scholars—or rather, the intuitions of those technologists and scholars who happen to agree with them, since there is certainly no consensus in the field. 

There is no particular reason to think this type of reasoning can accurately predict breakthroughs for any technology, and it has not accurately predicted past breakthroughs, so there is no substance to any of these “timeline” predictions. The upshot is that, while there are internally-consistent philosophical arguments that AGI is possible, there is no good argument for when it might be built. Some of the AI Doomers object that they can dodge the necessity of justifying their views on timelines by stating their timelines in the form of a probability distribution, but of course this does not work, since such a probability distribution is nothing but a way of expressing a particular viewpoint, which must still have been produced by reasons—perhaps good reasons, perhaps bad reasons.

It’s worth noting that the sharpest AI Doomers, those closest to the spring of the intellectual waterfall, are not strongly wedded to “short timelines.” Among this crowd, the dominant view is that AGI takeover is fairly likely to happen soon, and this is a sufficient reason for extreme action, but that the strong confidence and drive towards quantification among the mass of followers is unjustified. Eliezer Yudkowsky, the father of the movement, has spoken against “AI timelines,” among other notables.

As an aside, this is a common split in the beliefs of utopian-apocalyptic movements. The leading thinkers hold that the world will be irrevocably transformed sometime, which might come soon or late, and are acutely aware that it might not be imminent. Among the early Christians, the Apostles themselves were careful to specify that no one could know exactly when Christ would return, but emphasized that it might happen any day. It was the mass of their followers who expected the Second Coming in their own lifetime. Similarly, in the 19th and 20th centuries, Marx himself and the leading communist intellectuals claimed only that communism was eventually inevitable, and it was their followers who confidently expected to see the end of capitalism themselves.

The leading intellectuals of the 20th century Gaian environmentalists, in works like The Limits to Growth, were careful to specify that their mathematical models showing the imminent collapse of industrial society were mere illustrations and possibilities of an eventual exponential trend, but enough of their followers took them at face value to implement China’s one-child policy and India’s coercive sterilization program, while in freer nations many families voluntarily had fewer children. Of course these psychological patterns in how people and movements react to utopian-apocalyptic eschatologies do not directly bear on whether any particular utopian-apocalyptic claim is true or false, but these patterns are worth understanding anyway.

Society’s Response to Artificial General Intelligence Does Not Improve Over Time 

AI Doomer groups agitate for an “AI Pause,” or other methods of slowing AI development and “buying time.” The argument they give is that humanity is doomed by default because research to make AI more powerful progresses faster than research to align AI with human values. The current structure of society makes this all but inevitable, so if we are to avoid doom, then new policies are necessary, although no one has yet articulated what policies would produce an aligned AGI. Regardless, if the critical moment occurs in, say, 2032 instead of 2027, that gives more time for the world in general, and the U.S. in particular, to get its act together rather than following the natural incentives to carelessly build an unaligned superintelligence.

This is a strange and unjustified assumption! I would not say that the U.S. has been getting its act together, lately. In fact it seems like it’s becoming more and more of a mess as time goes on. It seems clear that an Obama administration would’ve done a better job at navigating the deployment of a new world-changing technology than a Biden administration, and an Eisenhower administration would’ve done a better job than an Obama administration. In ten years we’ll probably look back and long for the relative functionality of the Biden administration and lament that our government is now less able to deal with complex problems. The overall decline in competence is no longer a controversial position, so I won’t relitigate that at length. The upshot is that, so long as this goes on, kicking the can down the road by five or ten years makes things worse.

Personally, I don’t think this is especially worrying with regard to AGI, because I don’t see any reason to think AGI is coming soon—though I’m pretty worried about it for other reasons! The functionality of any social system waxes and wanes, with brief sharp spikes where new functional institutional templates are designed and propagated, then long periods of gradual decay, until the next functional form is created internally or imposed from without. Or, if you don’t want to get into how all the cycles work, then hopefully you’ll at least agree with the more general claim that sometimes a society gets more functional and sometimes it gets less functional, and lately ours has been getting less functional.

Assuming superintelligent computers are coming at all, I see no good reason to expect they will come during the current cycle rather than the next one, or two cycles from now, or five, or twenty. It certainly does matter whether AGI comes during a peak of functionality or a trough of incompetence, but without a justified model of when it’s coming—and I will repeat ad nauseam that nobody has one of those—interventions to change its timing are fumbling in the dark. An arbitrary delay could push the invention of AGI into a less competent period as easily as it could push it into a more competent period. Figuring out whether a delay is good or bad requires specific knowledge of both AI timelines and of changes in institutional functionality. “AI Pause” proponents have neither.

But if I were somehow convinced that AGI was coming in the current cycle—and again, there is no good reason to think this—then I would want it to come sooner, before the decay metastasizes even further.

Trying to make it more likely that the superintelligence comes at a peak of institutional functionality makes sense as far as it goes, but rather than trying to make the presumed-inevitable creation of AGI come at a particular moment, this can be done better by improving the health of the relevant institutions in general. Government representatives and bureaucrats capable of understanding the fields they regulate, universities purged of rampant fraud and quackery, technology corporations led by ascended engineers who understand their own production process rather than being driven into the ground by financiers—if AGI happens to come in the near future, any of these would make it far more likely that society responds competently. And if it happens to be further away than that, or to be unachievable for some as-yet-unknown reason, then improving our core institutions will have been a much better use of a career than kneecapping beneficial technology over fears that never materialize, like the overpopulation degrowthers or the anti-nuclear-power activists. If everything depends on our competence at an unknown critical moment, then it is better to build competence in our own time just in case it comes soon, rather than blindly try to match a completely unknown timeline for the arrival of AGI to an at-best-vaguely-guessable timeline for institutional functionality.

Unless, of course, you plan to take power personally, and need more time to do it. From that perspective, slowing things down becomes coherent.

Your AI Safety Coalition is Not Special 

The remaining justification AI Doomers give for why the societal response to AGI improves over time is that, over time, the AI Doomer coalition will gain power, and that, once in power, it will be uniquely able to make sure the superintelligent computer creates a utopia rather than exterminating all mankind.

The justification for expecting the AI Doomers to gain in power is straightforward and solid. When an organized group of young ideologues of the right skills and background sets their mind on gaining power and then works at it diligently for decades, they frequently succeed. And in fact the AI Doomers have done very well at taking power. Most notably, the household of husband and wife Holden Karnofsky and Daniela Amodei has overseen the disbursement of billions of dollars of charitable donations through GiveWell and the Open Philanthropy Project, including approximately all funding for all Effective Altruist organizations, and hundreds of millions for AI R&D and regulation, and separately controls Anthropic, a leading AI developer which created the Claude series of chatbots.

Elsewhere, Jason Matheny, a longtime Effective Altruist who researched existential risk at Nick Bostrom’s Future of Humanity Institute, has risen through the U.S. security state and currently leads the RAND Corporation. In 2023, the Open Philanthropy Project gave RAND $10.5 million for “AI governance” work, “to be spent at the discretion of RAND president and CEO Jason Matheny.” He was briefly a trustee of Anthropic’s Public Benefit Trust from its 2023 announcement until he stepped down months later “to preempt any potential conflicts of interest that might arise with RAND Corporation’s policy-related initiatives.” Many other activists are also doing increasingly well, many of them through the plethora of small Open Philanthropy-funded “AI policy” organizations that provide activists with a stable career track, which Open Philanthropy directed about $15 million to in 2024 alone. These organizations have an increasingly large hand in writing AI legislation, especially for individual U.S. states, and likely soon for European nations.

While their trajectory to power seems solid, the assumption that, once in power, this coalition will be especially able to steer society to develop an aligned AGI, or at least prevent the creation of unaligned AGI, is much more tenuous. One might justify such a claim, firstly by arguing that they have a strong track record of steering society thus far; or secondly, by arguing that the policies they advocate for are especially good; or thirdly, by arguing that they are intrinsically disposed to make better decisions. None of these are true.

Firstly, the AI Doomers’ track record at steering the course of AI development is abysmal. Over two decades of advocacy, they have caused exactly the fate they claim they want to prevent, via a series of own goals which can only be compared to environmentalist opposition to nuclear power causing the construction of coal-burning power plants. Who has accelerated AI development more than the AI Doomers who speak in dire tones of the importance of slowing it down?

Of the recent wave of AI companies, the earliest one, DeepMind, relied on the Rationalists for its early funding. The first investor, Peter Thiel, was a donor to Eliezer Yudkowsky’s Singularity Institute for Artificial Intelligence (SIAI, but now MIRI, the Machine Intelligence Research Institute) who met DeepMind’s founder at an SIAI event. Jaan Tallinn, the most important Rationalist donor, was also a critical early investor. Later, OpenAI was founded by a team that was explicitly inspired by Yudkowsky’s arguments about the power and possibility of computer intelligence, and more immediately by a conference held by the Future of Life Institute (FLI), an Open Philanthropy-funded organization created to popularize Yudkowsky’s views on AI.

At this conference, Elon Musk and Ilya Sutskever signed an open letter on the importance of AI safety, which would prove to be the first of several similar letters; months later, the pair were among OpenAI’s cofounders. Both later left OpenAI. Musk left in 2018, allegedly due to conflict of interest with his AI development at Tesla Motors, and Sutskever was forced out months after his participation in the AI Doomers’ failed attempt to eject CEO Sam Altman in 2023. Leading AI Doomers have since said that they regret their role in launching these AI labs.

In 2017, the Open Philanthropy Project directed $30 million to OpenAI, which was a nonprofit organization at the time. In return, Karnofsky received a seat on OpenAI’s board. In 2021, the brother and sister team of Daniela and Dario Amodei, Karnofsky’s wife and his roommate respectively, left OpenAI to found Anthropic, a rival AI company, citing Rationalist AI alignment ideas as justification. Anthropic soon received a $500 million investment from FTX, whose CEO Sam Bankman-Fried was the second-largest Effective Altruist donor until his arrest for fraud. Karnofsky resigned from OpenAI’s board due to the conflict of interest, and his seat was taken by Helen Toner, an inner-circle Effective Altruist and former Karnofsky employee, who later participated in—and may have instigated—the failed attempt to eject Altman.

The AI Doomers have also built a recruiting pipeline which sources, inspires, and sometimes trains a substantial fraction of the top research talent for all of these AI labs. This was done with the full and explicit backing of Rationalist and especially Effective Altruist leaders from 2015, when the leaders of AI labs were invited to speak and recruit at EA Global conferences, to 2022, when leading AI Doomers began publicly speaking against these labs. Since then the policy has become contentious, but it has not fully stopped.

That AI advocacy would accelerate AI development wasn’t some sort of unforeseeable surprise out of left field. It was entirely predictable. In fact many people predicted it. I was one of them, and my position was not considered novel or surprising at the time. In 2013, as the Rationalist campaign to raise awareness of AGI risk was taking off, I asked a couple of top Rationalist leaders—including the then-head of MIRI—why they thought their awareness campaign would help, given that it could very well inspire people to accelerate the research they spoke against. 

Both were clearly familiar with the “arms race” argument already, both said that inspiring development and making things worse was an entirely plausible outcome, neither gave any sort of argument that the benefits would outweigh the risks or struck me as especially interested in the question. In hindsight, the only flaw in the “arms race” argument is that it was too grandiose. We were imagining scenarios where “AI safety” researchers deliberately sprint towards AGI because of Space Race-level competition with credible foreign projects, not because of profit-driven competition between different American companies.

Will the same “raising awareness” strategy have different results now that they’re lobbying legislators instead of tech magnates? Of course not. If they succeed in persuading the U.S. government that AI is too powerful to leave unregulated, then the U.S. will respond the way it does to every powerful new technology—by funding and organizing its research and development. If and when the U.S. government takes charge of AI research, expect Berkeley Rationalists to start working for military labs that are explicitly working towards AGI while mouthing platitudes about “safety,” just like they did with all the previous AGI labs.

As for advocating specific policies, the AI Doomer coalition does not have a developed policy platform which they agree on. Its members vehemently disagree on how to solve the problems they organize around. Some support legal requirements that AI models must meet defined “safety benchmarks”, while others denounce this as “safetywashing” which creates a false and counterproductive impression that current techniques are fine without addressing the unique dangers of future superintelligent systems. Some AI Doomers support strong export controls and other policies to ensure Western countries maintain their lead in AI technology, especially over China, while others support conciliation and the creation of an international treaty and monitoring regime. 

Mostly they are bound together by the belief that Something Must Be Done, rather than any particular idea of what to do. Historically, this frequently prompts movements to push half-baked programs which satisfy members’ anxieties but do not achieve their strategic goals, and have a host of unintended consequences. It leaves a movement vulnerable to hijacking by opportunists who do not share their goals, and in fact this has happened to AI Doomers repeatedly—not just with the AI labs siphoning off researchers and funding, but also political opportunists, who have coopted the name and credibility of “AI safety” so thoroughly that the AI Doomers refused to be associated with the term they themselves coined. 

Because the AI Doomers lack a party line or any ability to police adherence, opportunists have been very good at wielding the resources assembled by AI Doomers, who have provided much of the funding and research talent for the AI research labs that they ostensibly oppose, and laid much of the intellectual and propaganda groundwork for the political factions who they treat as sometime-foes sometime-allies. The only policy that AI Doomers mostly agree on is that AI development should be slowed down somehow, in order to “buy time.” As discussed above, the reasons for this don’t hold together, except to justify their own quest for power—and like every political coalition in the history of the human species, they certainly agree about that. But unless one believes that the U.S. government of 2035 is likely to reverse its decline and be better able to react effectively to social problems than the U.S. government of 2025, the only remaining reason why slowing down AI development would be more likely to result in friendly AGI would be if the AI Doomers themselves simply have better character which makes them better able solve the problem even apart from any specific policies they might or might not pursue.

This was once a popular belief among the Rationalists. However, it has fallen by the wayside many years ago. 

The Old Movement’s Discarded Dreams

As the name suggests, the Rationalist community was founded on the promise of “refining the art of human rationality.” Readers today might be surprised that Yudkowsky’s essays, which created the group, have artificial intelligence as only a secondary topic, and are primarily about developing mental techniques for unlocking the full potential of human thought. This was the main focus of the movement in its early years, and Yudkowsky’s writing does indeed describe some useful mental practices which a diligent and introspective reader can practice and adopt. The hope was that the movement’s research into mental practices and insight into the laws of ideal thought would make them extremely competent in all walks of life. 

This belief was explicit in Yudkowsky’s Sequences essays, and inspired his followers to establish the Center For Applied Rationality (CFAR) to develop further techniques. As long as the Rationalists still mostly believed this would work, there was a justification for believing that their own people would govern better, especially on this topic where they had most thoroughly documented the irrationalities which ostensibly made it difficult for outsiders to act rightly. To make a long story short, the Rationalist training programs failed to deliver the promised upgrades, so the movement rationally updated their views in accordance with the evidence, and this belief was gradually walked back and abandoned. 

By about 2017, the movement had thoroughly discarded its early dream of serving as a vanguard party wielding the perfected techniques of reason, and of “raising the sanity waterline” by distributing these techniques to the population at large. CFAR’s original cadre of true believers left the organization and were replaced by less ambitious staff. Today, a minority of the Rationalist old guard still hold to the belief in the power of trained rationality techniques in their personal lives, but it no longer plays a part in Rationalist plans for political control of AI or for broader social change.

The Effective Altruists used to have a similar belief that they were inherently fit to rule, albeit with a different basis. While the Rationalists once believed that they wielded, or someday would wield, superior mental practices, the Effective Altruists once believed that they were uniquely willing to work in the wider interests of humanity rather than venal self-interest. They would frequently speak of a person’s “alignment,” a term originally from artificial intelligence theory used as shorthand for benevolent and impartial concern for all humanity, but soon repurposed to mean adherence to Effective Altruist professional backscratching and doctrine. Many Effective Altruists conflated the concepts, not only the terms. The term has continued spreading, and in recent months I’ve heard “alignment” used this way by unrelated social movements and elite university groups.

The Effective Altruist self-image was abruptly shattered by the 2022 arrest of Sam Bankman-Fried, Effective Altruism’s second-largest donor, for running one of the largest financial frauds in world history. This provoked a crisis of faith among Effective Altruists, and a great deal of rethinking. Their self-image as morally superior is now much more limited and healthy, reduced to about the level of any other political faction. The assumption that any Effective Altruist is ipso facto more benevolent and trustworthy than anyone else is no longer tenable, and no longer spoken in public.

Perhaps the changes in these movements are for the better. In any case, the less-ambitious self-appraisal has led them both to quietly withdraw these justifications for believing their members would govern AI better simply because of who they were.

Finally, there is also the position that all AI development should be halted indefinitely, to resume only after the creation and deployment of extremely powerful human intelligence augmentation technology, which would create researchers far smarter than any existing human being. This is advocated by Eliezer Yudkowsky, and as far as I can tell by no one else, although even for him the plan is sensibly “not being emphasized very hard in current messaging.” He hopes that once this project has engineered “[h]umans augmented far enough … that they never put faith in a step that will not hold,” the resulting specimens “can build ASI [artificial superintelligence] without making any hidden fatal mistakes”.

Of course, there is no indication that massive intelligence augmentation will be developed any time soon, only very weak reasons to suspect that it’s obtainable at all without multiple revolutionary breakthroughs in our understanding both of genetics and of the mind, and no reason at all to imagine that augmenting human intelligence would by itself instill the psychological changes towards humility and caution that Yudkowsky desires. But leaving aside the conflation of intelligence and epistemic humility, if we are overoptimistic to the point of pollyannaism, then we can imagine that it might take 25 years to create human intelligence augmentation technology which meets Yudkowsky’s standards for the creation of uplifted superhumans, and then 25 years for the newly-created supermen to grow up and embark upon their careers in AI research, for a total of a fifty-year ban on all AI research.

Of course, 25 years to superhuman intelligence augmentation would be shockingly, shockingly fast. While there is no firm ground for any prediction as to how long it will take before any technological breakthrough, if ever, it seems more likely that such a regime would have to last worldwide for a century or several centuries before such technology were created. Yudkowsky, presumably, is well aware that maintaining a worldwide ban on AI research for that long is nearly impossible, and would be historically unprecedented in several different ways, going far beyond, for example, existing restrictions on nuclear weapons in reliability, extent, duration, and most especially enforcement.

In Search of a Strategy That Might Work

The proposal’s impracticality notwithstanding, while the philosophical arguments for the speculative danger of AGI seem more than strong enough to justify technical research into the apparent problem, they are nowhere near a level of certainty strong enough to justify hundreds of years of panopticon global surveillance, banning “[a]nything you don’t need for computer game graphics,” and deigning to allow “small amounts of high compute [only] in datacenters under international supervision.” And of course, as algorithmic progress continues on whatever hardware is available, the bar for what someone can “safely” keep on a private computer would drop and drop and drop, even without considering how the datacenter bureaucrats would inevitably push for and receive more and more and more surveillance powers, requiring the recall and seizure of once-allowed computers.

The plan is a bad one, but it does have one very important virtue. The argument for the plan is at least locally valid, if you grant all of its absurd premises. In other words: if every single one of Yudkowsky’s many suppositions about hypothetical breakthrough technologies in fields as diverse as computing, genetics, and human psychology all turn out to be completely correct, and if the ridiculous impossibility of implementing the plan is overcome, and if we set aside the horrific costs to human liberty and to technological development, then the plan does at least help somewhat with the problem it claims to address. This makes it unique among political proposals for preventing the development of unaligned AGI.

The usual AI Doomer plan does not even make it this far. Even if all of their technological assumptions are correct, the plan of establishing progressively more onerous regulations so that bureaucrats can hobble AI research in hopes that this would “buy time” would not result in a better chance of creating an aligned AGI. Instead it would just shift the same AI research from privately-owned companies to the U.S. government, most likely via the military, most likely employing the same individual researchers as today.

The AI Doomers’ plans are based on an urgency which is widely assumed but never justified. For many of them, the urgency leads to a rush to do something—anything—even if their strategy is unsound and has historically done exactly the opposite of what they profess to want. In the absence of a specific policy proposal which is likely to work, those who accept the premises of AI Doom would do better to work on improving institutional functionality across the board, rather than attempting to “buy time” in the hopes that our declining institutions will unaccountably become more able to handle their concerns, rather than less able. Otherwise, their current program is on track to once again accelerate the research they hope to prevent.