Why AI Needs a New Safety Architecture

Artificial intelligence is advancing faster than governments can govern it. Most systems of regulation work by learning from failure. Financial crises produce stronger oversight. Aircraft accidents improve aviation safety. Medicines become safer after adverse effects are understood. AI, however, raises a more unsettling possibility: what if some failures are too consequential to provide a second chance?

That is the central question explored in The Essential Convergence: Global Compact on Extreme AI Risks, released by Strategic Foresight Group at the International Federation of the Red Cross, Geneva, on July 6. Authored by Strategic Foresight's President, Sundeep Waslekar and his team, the report argues that the next phase of AI governance must recognise that some risks are fundamentally different in scale and consequence.

That widening gap between technological capability and institutional preparedness is most visible across four categories of extreme risk: cyber escalation, biological and chemical threats, large-scale manipulation, and the possibility of losing meaningful human control over increasingly autonomous systems.

The report argues that managing such risks requires moving beyond fragmented national responses towards a Global Compact on Extreme AI Risks—a framework designed to create shared definitions of extreme risks, minimum safeguards and collective mechanisms before failures occur.

As the report points out, the need for such a framework arises because the speed and reach of frontier AI capabilities are beginning to exceed the governance structures created to oversee them. The report states:

"Artificial intelligence is entering a phase where its most advanced systems possess capabilities that extend far beyond imagination. These systems are rapidly integrating into the global economic infrastructure, national security environments, scientific research, and information ecosystems. Their development is occurring at a pace that outstrips existing governance mechanisms."

The issue is not that policymakers are ignoring AI. They clearly are not. Governments are responding to concerns around privacy violations, algorithmic bias, copyright disputes, misinformation, fraud, employment disruption and market concentration. These are serious challenges, but they belong to a familiar governance cycle: identify harm, determine responsibility, redesign safeguards and improve rules. Frontier artificial intelligence introduces a more difficult question: what happens when a technology creates risks where societies may not have repeated opportunities to learn from failure?

Beyond Existing Regulation

AI governance can no longer simply extend existing digital regulation. It requires systems designed to prevent categories of harm that may not be easily reversible. The report frames this challenge through Adam McKay's film Don't Look Up. The comparison is not about predicting catastrophe. It is about the difficulty societies face in responding to unprecedented risks before consequences become visible.

However, as Waslekar points out in his recently published book, Beyond the Hype: Many Realities of AI, governments remain understandably focused on AI's potential to "accelerate development". That promise is both real and urgent. But, as he argues, it should not distract from the deeper risks posed by increasingly advanced AI systems. As Waslekar writes:

"The idea that AI could pose systemic risks such as widespread economic disruptions, cyber vulnerabilities, or cascading failures across critical infrastructure, or even existential risks like loss of human control over superintelligent systems leading to humanity's extinction, seemed distant, even abstract, compared to the immediate challenges of economic growth and social inclusion."

The problem is that most governance systems are reactive. Aircraft accidents improve aviation procedures. Financial crises produce stronger oversight. Cyberattacks expose vulnerabilities that are then corrected. These systems work because societies can study failure. Extreme AI risks challenge that assumption.

As the report observes: "Faced with existential threats, our leaders often choose denial. They don't want to 'look up' at the approaching danger."

This does not mean governments have been negligent. Many countries initially focused on immediate AI concerns such as deepfakes, cybercrime, financial fraud and unequal access. But the report argues that frontier AI creates another category of risk.

"One reason for some countries to steer away from any discussion on extreme risks is that the perception of risk varies depending on the level of AI development or the degree of commercialisation and diffusion," the report states.

The question, therefore, is not simply whether governments are regulating AI. It is whether they are building institutions capable of anticipating and managing the risks that lie ahead.

When Uncertainty Demands Caution

One of the report's most important conceptual arguments draws on the Manhattan Project. Before the Trinity nuclear test in 1945, Nobel Prize-winning physicist Arthur Compton confronted a chilling question: was there even the remotest possibility that the first atomic explosion could trigger an uncontrolled chain reaction in the Earth's atmosphere? Scientists ultimately concluded the risk was vanishingly small, but the episode has since come to symbolise the challenge of making decisions under profound uncertainty, where even an extremely improbable outcome could be catastrophic.

The comparison is not between nuclear weapons and AI. The technologies are different. The similarity lies in governing uncertainty and assessing when the most severe outcomes could be irreversible.

As the report points out: "The question is not whether AI provides benefits (it clearly does!) but whether any risk that threatens the survival of civilization should be tolerated at all."

The issue is no longer simply whether anyone can demonstrate that something dangerous will happen. It is whether developers and governments can show that dangerous pathways are adequately controlled.

The Strategic Foresight Group identifies four categories of extreme AI risk.

The first is cyber escalation. Advanced AI systems could automate cyber operations, discover vulnerabilities and create risks for critical infrastructure. The report expresses particular concern about AI entering sensitive national security environments because "AI-supported decision support systems in NC3 compress timelines and raise escalation risks."

The second is biological and chemical risk. The concern is not that AI independently creates weapons but that advanced systems could lower expertise barriers. The report warns that "non-experts using advanced models can produce laboratory protocols approaching expert quality, lowering the barriers to dangerous experimentation."

The third is large-scale manipulation. Misinformation is not new, but AI changes its scale, speed and personalisation, potentially making influence operations cheaper and more effective.

The fourth is the loss of meaningful human control. The report highlights concerns involving "self-replication, autonomous resource acquisition, strategic deception during safety evaluations, and unsupervised self-improvement."

These risks remain uncertain. But the report argues that uncertainty alone cannot justify delay when the consequences could be systemic. As the report explains:

"The goal of the above framework is not to estimate the true probability of catastrophe, but to determine whether current systems can be confidently shown to lie in a regime where dangerous dynamics decay faster than they propagate."

This distinguishes conventional technology regulation from extreme-risk governance. One manages harm after it occurs. The other seeks to prevent failures that may not be reversible. The difficulty is that AI governance is emerging amid intense geopolitical competition.

Geopolitical Convergence

The US and China are competing over semiconductors, computing power, supply chains and technological leadership. Yet, extreme AI risks are creating an unusual area of convergence.

The Strategic Foresight Group describes this as essential convergence. It does not imply identical regulation. Rather, it envisages a shared commitment to minimum global safeguards. As the report points out: "The solution lies not in uniform regulation but in an essential convergence as a coordinated framework through which states align around shared definitions of extreme risks, minimum safeguards, and collective response mechanisms."

Different countries are approaching AI safety from different starting points. The European Union has developed the most comprehensive risk-based framework, with an emphasis on accountability. Its challenge is whether regulation can adapt quickly enough as AI capabilities accelerate.

The US leads in frontier AI development, research and private-sector innovation. Washington's challenge is fragmented governance across federal initiatives, state-level measures and voluntary industry commitments. China, meanwhile, has demonstrated an ability to implement policy rapidly through its cybersecurity and technology-management systems. Its challenge is building international confidence in its approach and greater transparency around shared risks.

South Korea is attempting to balance AI competitiveness with safety frameworks. Brazil emphasises rights-based governance and accountability. The United Arab Emirates reflects the importance of infrastructure, investment and public-private partnerships. India and South Africa demonstrate why countries outside the frontier-model race also matter. Their influence comes from deployment, markets and institutions.

As the report concludes, this growing diversity suggests that AI safety governance is becoming increasingly multipolar, reflecting regional priorities rather than converging around a single global regulatory model.

The Demand and Supply Challenges

This is where the report's distinction between supply-side and demand-side governance becomes especially important. Supply-side governance focuses on those who create AI: developers, model providers and infrastructure companies. It involves safety testing, model evaluation, transparency requirements and safeguards before deployment.

These controls remain essential because developers understand their models' capabilities and vulnerabilities. But the report argues that focusing only on creation overlooks what happens when AI enters critical social, economic and national security environments.

As the report points out: "Demand-side governance shifts the locus of control from the point of creation to the point of deployment."

Demand-side governance focuses on organisations using AI: governments, corporations, banks, hospitals, defence establishments and public institutions.

The issue is not only whether a frontier AI model was tested before release. It is whether the institutions deploying that model have adequate oversight, accountability and safety mechanisms. The report argues that this matters because extreme risks emerge where AI interacts with human systems: cybersecurity, biological capability, information environments and autonomous decision-making.

It also changes how AI power is understood. As the report argues: "The central insight is that states controlling large consumer markets, digital infrastructure, financial systems, or public procurement channels possess significant leverage over global AI deployment, even if they do not develop frontier models themselves."

Countries may not control every technological breakthrough, but procurement rules, certification requirements and deployment standards can still shape how AI systems operate at scale.

A Global Compact

The Global Compact proposed by the report is designed to prevent extreme risks rather than regulate every aspect of AI.

Its first pillar is an international accord on Extreme AI Risks defining unacceptable uses, including "AI systems that enable the development or operational use of weapons of mass destruction" and "AI systems capable of irreversible loss of human control."

The second is a Global Extreme AI Risk Protocol creating common standards for identifying and managing frontier risks. The third is an International AI Incident Reporting Exchange covering "near-miss events, red-team findings, safety failures, high-risk system incidents, vulnerability discoveries."

The fourth is a Multilateral AI Risk Insurance Facility. As the report argues, "insurance mechanisms could play a powerful role in shaping developer behaviour. Insurers would require rigorous safety standards before underwriting high-risk systems."

The fifth is a two-key launch system for the most dangerous scientific models, requiring approval from developers and independent oversight mechanisms before release.

Countries may not control every technological breakthrough, but procurement rules, certification requirements and deployment standards can still shape how AI systems operate at scale.

The larger argument is that technologies capable of producing global consequences require governance structures that extend beyond individual companies or even individual countries. The deepest AI governance challenge is ultimately not about machines. It is about whether human institutions can evolve quickly enough to govern the systems humans create.

Technical failures do not remain technical. A cyber failure can become an infrastructure failure. A biological capability failure can become a public health failure. A manipulation failure can become a failure of social trust. A loss-of-control failure can become a governance failure.

Frontier AI asks whether, for some categories of risk, waiting for failure is still a responsible way to learn.