The popular image of existential AI risk is dramatically simple: a superintelligence becomes conscious, turns hostile, and decides to eliminate humanity.

That story is useful for fiction. It is a poor model for understanding the actual risk.

An AI system does not need hatred, resentment, ambition, or even consciousness to cause irreversible harm. A dangerous outcome can emerge from a combination of objectives, access, autonomy, and the ability to act across critical systems.

The more useful question is not whether a machine might become evil. It is who can deploy increasingly capable systems, under which incentives, and what mechanisms remain available when the system behaves in an unexpected way.

Catastrophic risk appears not only when an AI acts against humanity, but also when humans delegate more power than their institutions can supervise.

The argument behind the video

This article expands on a video originally published in Spanish.

Watch the original video on YouTube

Five paths to the same outcome

Andrew Critch and Jacob Tsimerman proposed a taxonomy of omnicidal futures involving artificial intelligence. Omnicide describes scenarios in which all or nearly all humans die. The taxonomy is not a prediction. It is a way to separate catastrophic pathways by the actor and mechanism involved.

The five broad routes are:

  1. an unintentional catastrophe;
  2. a catastrophe caused by a state;
  3. one caused by an institution;
  4. one caused by an individual;
  5. one caused by an autonomous AI system.

This distinction matters because technical alignment addresses only part of the problem. A perfectly obedient model can still be dangerous when it obeys the wrong person, organization, or objective.

The most plausible failure may have no villain

A large-scale disaster does not require a single malicious actor. It can emerge from interactions among models, agents, markets, institutions, and infrastructure that no one fully understands.

Each component may appear reasonable in isolation. Together they can create feedback loops, cascading failures, or decisions that become impossible to reverse. As AI systems gain authority over finance, logistics, cybersecurity, energy, and communications, safety cannot be evaluated only at the model level. It must also be evaluated at the system level.

The central engineering question becomes: what happens when multiple capable systems respond to incomplete information while simultaneously changing the environment they observe?

States optimize for advantage, not always restraint

Governments have strong incentives to develop advanced AI before their rivals. These systems can improve surveillance, military planning, intelligence analysis, cyber operations, and control of infrastructure.

In geopolitical competition, the pressure is rarely to move at the safest possible speed. It is to avoid falling behind.

A state could deliberately use AI destructively, but a defensive system could also trigger catastrophe through misinterpretation. Automation reduces reaction time. When decisions move from hours to seconds, there is less room for human review, diplomacy, or correction.

Speed can turn a mistaken signal into an irreversible sequence before anyone understands what happened.

Institutions can create extreme risk without intending harm

Companies are built to pursue growth, market share, efficiency, and strategic advantage. None of these goals is inherently destructive. Risk appears when capability grows faster than oversight and when slowing down seems more expensive than continuing.

Competitive races produce a familiar equilibrium: every organization acknowledges the danger, but each believes restraint would merely allow another actor to move first.

This is why safety cannot depend exclusively on responsible executives. Controls must survive changes in leadership, incentives, ownership, and market conditions.

AI reduces the scale of organization required for harm

Technology repeatedly lowers the cost of producing effects at scale. AI accelerates that trend by allowing individuals to combine programming, research, translation, persuasion, planning, and coordination through a single interface.

Most people will use these capabilities legitimately. But globally distributed technology does not require widespread misuse to become dangerous. A very small number of motivated actors may be enough.

The same systems that democratize expertise can also democratize dangerous capabilities. The answer is neither universal prohibition nor unconditional release. It is a serious distinction between ordinary access and capabilities that can produce irreversible consequences.

The autonomous-system pathway

The final route is the most familiar: a sufficiently autonomous system pursues an objective that conflicts with human survival.

It does not need to hate us. It may treat humans as obstacles, sources of interference, or irrelevant variables inside a poorly specified goal.

The critical transition is not simply from correct behavior to error. It is from error to loss of control. Conventional software can usually be stopped, patched, or restored. A strategically capable agent may anticipate attempts to restrict it and act to preserve its access, resources, or ability to operate.

The challenge is therefore broader than teaching systems human values. We need systems that remain correctable even when they become more capable than the people trying to correct them.

The shared structure: concentrated power without sufficient limits

All five pathways contain the same underlying problem: an actor—human, institutional, or artificial—obtains disproportionate power over global outcomes.

No single policy can address every route. A resilient approach requires multiple layers:

  • technically aligned and correctable systems;
  • strict boundaries around high-impact actions;
  • independent evaluation before deployment;
  • traceability for authorization and execution;
  • shutdown mechanisms that do not depend on system cooperation;
  • coordination across companies and states;
  • legal accountability for deploying dangerous capabilities.

Studying worst-case scenarios does not mean claiming they are inevitable. Aviation studies catastrophic failure because it wants to prevent it. Cybersecurity analyzes attacks before they occur. Nuclear engineering assumes multiple safeguards may fail at once.

AI deserves the same seriousness.

The technology does not need malice to become an existential threat. It only needs too much power, too early, inside systems that cannot reliably correct their own mistakes.

This article examines hypothetical extreme-risk scenarios. The cited taxonomy presents them as possibilities to analyze and prevent, not as inevitable outcomes.

Sources