What happened
Anthropic proposed a new regulatory framework to California Governor Gavin Newsom that would allow government officials to petition courts to block the release of AI models if testing indicates they pose a serious risk to society. The company also recommended against mandatory 'kill switches' for entire models, advocating instead for targeted measures like shutting down specific agents or terminating client access. This proposal follows the formation of an AI expert group by Newsom in September to draft safety requirements.
According to Channel NewsAsia, Anthropic submitted a letter dated October 15 to California Governor Gavin Newsom. The letter proposes that government officials be granted the authority to go to court to seek orders blocking the release of artificial intelligence models. This action would be triggered if testing finds that a specific model poses a serious risk to society. An Anthropic representative confirmed the authenticity of the letter, while the California governor's office did not provide an immediate comment.
The proposal was submitted after Governor Newsom formed a group of artificial intelligence experts in September. This group was tasked with preparing proposals on requirements. Anthropic's submission appears to be a direct response to this initiative, offering specific mechanisms for implementing safety controls.
In the same letter, Anthropic opposed a requirement to build 'kill switches' into AI models that would allow for the complete cessation of a system's operation if the company loses control. The company argued that targeted measures would be more effective. These measures include shutting down a problematic or terminating a specific client's access to the model, rather than disabling the entire model infrastructure.
This development occurs in the context of broader U.S. legislative efforts. In September, a bipartisan group of U.S. senators discussed a bill that could give the government authority to block the release of individual AI models deemed dangerous, with companies retaining the right to challenge such decisions in federal court. However, Congress did not advance this proposal further after the election recess. Additionally, in July, members of the U.S. House of Representatives introduced the AI Kill Switch Act, which would provide officials with the right to require AI companies to stop models that pose a risk to human life or the economy.
Why it matters
This proposal represents a significant shift in how governance might be structured in the United States, moving from voluntary industry standards or broad legislative mandates to a judicial oversight model. By suggesting a court-based blocking mechanism, Anthropic is engaging with the legal infrastructure for AI risk management, potentially setting a precedent for how high-risk models are regulated. The rejection of blanket kill switches in favor of targeted interventions highlights a technical and operational debate about the feasibility and effectiveness of different safety controls. This move occurs amidst ongoing legislative efforts, such as the AI Kill Switch Act, and previous executive actions regarding model access, indicating a complex and evolving landscape for AI regulation.
The proposal introduces a judicial check on AI model deployment, which is a novel approach compared to traditional regulatory or legislative mandates. By involving the courts, the process may offer a more nuanced and case-by-case assessment of risk, potentially reducing the burden on executive agencies to make technical determinations about model safety.
Anthropic's opposition to mandatory kill switches highlights a practical concern in AI engineering. Implementing a global kill switch for a complex model may be technically difficult or could have unintended consequences for other users or applications. Targeted interventions, such as disabling specific agents or user accounts, may be more precise and less disruptive, aligning with the company's operational capabilities.
This move by Anthropic, a major player in the AI industry, signals a willingness to engage with regulatory frameworks that include potential restrictions on model release. It may influence the debate on by providing a concrete example of how industry and government can collaborate on risk mitigation strategies.
The timing of the proposal, following the formation of an expert group by the California governor, suggests that state-level initiatives are playing a significant role in shaping AI policy, especially as federal legislation remains stalled. California's regulatory environment often sets a precedent for other states and can influence national standards.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
crm_get_transaction(id='4092').Why can ethical evaluation not be reduced to one model score?
What to watch next
Monitor the California governor's office response to the proposal and any subsequent legislative or executive actions in California regarding AI model release controls. Watch for reactions from other AI companies and regulatory bodies to this proposed judicial framework. Track whether this proposal influences the stalled bipartisan Senate bill or the House AI Kill Switch Act, potentially shaping the final form of federal legislation.
The response from Governor Newsom's office and the AI expert group will be crucial in determining whether this proposal will be adopted or modified. Any public statements or policy documents released by the governor's office will provide insight into the state's direction on .
Reactions from other AI companies, such as OpenAI and Google, will indicate whether this judicial blocking mechanism is seen as a viable industry standard or a potential barrier to innovation. Their public statements or lobbying efforts may shape the final policy outcome.
The status of the bipartisan Senate bill and the House AI Kill Switch Act will be important to monitor. If these federal bills gain momentum, they may incorporate elements of Anthropic's proposal or offer alternative frameworks for AI model regulation.
Legal experts and researchers may provide analysis on the feasibility and effectiveness of court-ordered model blocks. Their insights will help assess the practical implications of this proposal for the AI industry and public safety.