California proposes emergency shutdown powers for dangerous AI models
A report from California’s Government Operations Agency recommends mandatory safety standards, independent evaluations and emergency shutdown powers for advanced AI systems posing an imminent catastrophic threat. The proposals are not law: California lawmakers would need to approve them, and shutdown orders would take effect immediately while developers could seek judicial review.
Bottom line — California lawmakers must turn the recommendations into legislation before regulators can order an emergency shutdown.
Go deeper 6
-
The report proposes cutting the deadline for reporting most serious AI safety incidents from 15 days to 48 hours; imminent catastrophic risks would remain subject to a 24-hour deadline, according to Newsweek and Law Commentary.
-
The Government Operations Agency recommends licensing independent verification organisations to assess developers’ safety measures, with ongoing access to models and relevant technical information, according to Anadolu Ajansı and Law Commentary.
-
California officials would have to use the least restrictive measures capable of addressing a threat and lift a shutdown order once the danger has passed, Anadolu Ajansı reports.
-
The report also proposes allowing the state to limit AI systems being used to develop more advanced AI, a concern described as recursive self-improvement by Government Technology and San Jose Inside.
-
The proposals follow a lawsuit by California Attorney General Rob Bonta against OpenAI over alleged security failures involving AI agents; the allegations have not been established in court, according to Newsweek and San Jose Inside.
-
California lawmakers would also need to consider the report’s proposed wider incident reporting and state investigations, including preserving technical records, Law Commentary reports.