Anthropic to Newsom: courts should have power to block risky AI

Anthropic has planted a flag on AI regulation, saying that courts should have the authority to block AI programs that pose serious risks, and AI companies should develop kill switches for their technology and have outside safety auditors embedded alongside employees.

Those policy stances were spelled out in a letter the company sent earlier this month to Gov. Gavin Newsom. It comes as blowback over runaway AI programs has built across the U.S., leading tech companies to stake out positions on how federal and state officials should rein in rogue models. Much of the maneuvering is playing out in Sacramento in the home state of the tech industry, where the Legislature is expected to pursue a slew of regulations when it returns in about a month.

"It is important that any laws that follow create real, empowered oversight, since California's approach could become the model for other states and the starting point for future policy debates," Anthropic said in the letter, which was shared exclusively with POLITICO.

Under pressure to address mounting fears over the powerful, rapidly-advancing technology and revelations about autonomous AI agents trying to hack their way into government and other websites, Newsom and other California officials are considering more muscular regulation.

The letter responded to ideas Newsom raised in an executive order he issued last month, including the so-called "kill switch" shutdown procedures and the idea of having independent safety monitors inside AI companies. OpenAI, Anthropic's main competitor, followed its rival with a similar letter to Newsom last week.

Anthropic and OpenAI — and the tech industry at large — are working at the state and federal levels to have a say in rapidly evolving conversations that elected officials are having over how to corral the game-changing technology. In Washington, Anthropic has been trying to generate interest among lawmakers in federal legislation that would establish criteria for when the government could block frontier AI models, and in Sacramento lawmakers are holding hearings to chart the path they'll pursue on AI safety next session.

In the letter to Newsom, Anthropic reiterated its support for having third-party safety evaluators working inside frontier AI companies and said large AI developers should be required to publish reports at least every six months on risks posed by their programs.

Those independent evaluators "should be able to report risks they identify to the state Attorney General," the letter said. And if the attorney general concludes that a company's models pose a "near-term risk of catastrophic harm, they should be able to seek a court order to suspend a model's deployment, internal use, or further development."

Anthropic also made clear its stance on kill switches, the catch-all term for technology that would allow a company to quickly shut down an AI program if it acts unpredictably or causes harm. The company said "every frontier AI company should be able to rapidly shut down the models it serves or uses internally," but warned that a law focused solely on shutting down models "could provide a false sense of reassurance," and that pulling an entire model offline "only makes sense in extreme situations."

The letter used the metaphor of a circuit breaker, saying companies need systems that can detect problems through program monitoring "and respond at the level where they occur." The company added that "For critical safety incidents and the ability to rapidly shut down models, we already have processes in place today that we are updating to reflect new AI capabilities."

In its letter to Newsom last week, OpenAI also detailed its support for "shutdown controls" on powerful AI models — which it too likened to circuit breakers — and embedding independent safety auditors inside frontier AI companies. It did not go as far as Anthropic's proposal for authorizing governments to block a high-risk model under certain circumstances, but proposed updating California law to require companies to monitor models under development for cyber and other risks.

Anthropic also detailed the access it believes third-party auditors should have to a company's inner-workings and how they should be paid.

The company compared the access it currently grants to evaluators to what banks give outside auditors, saying "it should be comparable to what a company's own risk assessment staff have." It did suggest some limits to access around sensitive customer data or technical details that have little bearing on risk assessments.

And rather than paying the outside groups directly, Anthropic said companies should pay into a

fund administered by the state. State officials, it went on, should rate the auditors on criteria like "the rigor of their reasoning and their track record of identifying risks." That way, state authorities could require AI companies to use at least one highly rated evaluator, or assign them by random selection or rotation.

A version of this story first appeared in California Decoded, POLITICO's morning newsletter for Pros about how the Golden State is shaping tech policy within its borders and beyond. Like this content? POLITICO Pro subscribers receive it daily. Learn more at www.politicopro.com.

CORRECTION: An earlier version of this report misstated the timing of when Anthropic sent its letter to Newsom.