Recent discourse from major AI labs like OpenAI and Anthropic has increasingly centered on the existential threat posed by super-intelligent AI. These organizations are grappling with the daunting task of ensuring that a super-intelligence - an entity with cognitive capabilities far exceeding our own - remains “aligned” with human values. And sure, when a random job hopping employee gets prime time for claiming there’s a 10% risk of human extinction within a few years what do you expect?
I’m writing this post because I see lots of people saying that yes, sure, let’s move forward slowly. After all, we didn’t have LLMs (the “AI” in question) a few years ago so why not make some global agreement - like the Montreal Protocol on banning CFCs, which remains one of the most successful environmental treaties in history (UNEP, 2023 ), or the Non-Proliferation Treaty on nuclear weapons (State Dept, 1968 ), and be done with it? Both of these have been resounding successes, stopping the growth of the ozone hole and with the exception of some rogue nation states nuclear weapons development has been kept in the hands of few.
However, there is a fundamental difference between the ozone crisis and the AI alignment problem.
The ozone layer is a “common pool” resource. If a single company in a remote corner of the world continued to use CFCs (which happened as recently as 2019 ), the impact on the global atmosphere would be negligible. “Good enough” is a reachable target. In contrast, the threat from super-intelligent AI is that it only needs to happen once. It only takes a single person, a single lab, or a single rogue actor to develop a capability that could fundamentally alter the human trajectory.
This mimics the existential threat of biohacking. Just as a small laboratory with a desktop sequencer and enough motivation could theoretically synthesize a lethal virus in a backyard (PMC, 2023 ), the democratization of high-quality weights and training techniques means that the barrier to creating a “doomsday” AI follows Moore’s Law. Because of the immense economic and geopolitical rewards associated with creating the “smartest” AI, we can be virtually certain that someone - someone with enough motivation and resources - will eventually succeed. We’ve just seen that terrorist organizations try to develop better weapons using the cloud LLMs they have access to today (AP, 2024 ). If we manage to put a global moratorium on the development of better LLMs, the ability to create one outside of that control will make it possible for bad actors to take a very real and assymmetric step forward.
This leads us to a dystopic realization. Unlike CFCs, which we could regulate by monitoring diffuse industrial outputs, AI development may require a level of oversight that seems right out of an 80s sci-fi.
If the threat is existential and the risk can come from anyone, the only viable solution is a global authoritarian regime. Such a regime would have the power to monitor and enforce strict limitations on the use of computer hardware, effectively capping the freedom to develop LLMs. Of course, you cannot just regulate the development of LLMs - you would need to inspect any and all development, for any stated purpose. It would be a system where the ultimate goal of human safety overrides the sanctity of private enterprise and individual liberty.
… and this is not just about the creation of super-intelligent AI. As I alluded to above, we face the very same threat when it comes to biohacking terrorists. I posit that the threat of engineered organisms wiping us out is a lot more urgent than AIs, simply because there are ways to shut down AIs that don’t exist for viruses.
The road to a global authoritarian state is paved with good intentions.
power off