Is regulation enough to stop AI systems killing us all? | Letters

THE NEWS

What happened

An expert letter questions whether current regulatory frameworks can ensure the safety of frontier AI systems that pose existential risks. The author argues that while aviation and nuclear industries use rigorous 'safety cases' to prove low accident probabilities, AI developers have provided no equivalent evidence for their systems, making independent oversight insufficient without concrete risk analyses.

CONTEXT

Why it matters

Is regulation enough to stop AI systems from causing existential harm? A recent letter highlights that while industries like aviation use rigorous 'safety cases' to prove low accident risks, frontier AI developers have offered no equivalent evidence. Withdrawing a model after internal failures shows the gap between current oversight and the extreme stakes involved.

AT A GLANCE

Key facts

• OpenAI recently withdrew a new frontier AI model after failing internal safety tests.
• Industry standards for safety-critical systems like aircraft require 'safety cases' demonstrating extremely low accident probabilities (e.g., less than once in 1,000 years).
• Developers of frontier AI systems have not provided detailed risk analyses or independent safety cases despite claims that their systems could kill humanity.
• The letter argues that policy based on fantasy rather than rigorous analysis is dangerous for existential risks.