US legislators push AI safety laws amid human extinction warnings

Current and former artificial intelligence researchers in the United States have issued dire warnings that the technology could soon lead to human extinction, capturing the attention of lawmakers in Washington.

The most recent warning came in a lengthy social media post by Jacob Coxon, a San Francisco-based researcher who announced his resignation from Anthropic on Tuesday evening.

“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear,” Coxon said in his post, which prompted a flurry of responses by AI experts also sounding the alarm.

Evan Hubinger, a current Alignment Science lead at Anthropic, echoed his remarks, saying, “we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to”.

Coxon declined Al Jazeera’s request for an interview. Hubinger did not respond.

How is Washington handling it?

The posts have sent a ripple effect through Washington.

On Wednesday, congressman Josh Gottheimer, a Democrat, and Mike Lawler, a Republican, introduced a bipartisan House bill aimed at preventing AI systems from operating on their own without human oversight.

The Stop Rogue AI Act would ensure that federal agencies have the tools and ability to spot dangerous AI systems running on their networks and shut them down before they can cause harm.

Independent Senator Bernie Sanders and House congressional Representative Greg Casar, a Democrat, also ramped up calls for their proposed legislation that would ban the development and deployment of artificial superintelligence. The legislation would also pause AI development until federal safety rules are put in place. Sanders is also reportedly convening a bipartisan briefing to address the elevated risks posed by AI.

On the other side of the political spectrum, Republican Senator Ted Cruz also voiced concerns about AI on ABC’s The View.

“We’ve got to put some guardrails on it,” Cruz said.

Cruz said he is working on bipartisan legislation with senators Amy Klobuchar, a Democrat, and John Thune, a Republican who serves as Senate majority leader, that would address instances of potential catastrophic harm.

The legislation is similar to legislation in the House of Representatives.

In July, representatives Ted Lieu, a Democrat, and Nathaniel Moran, a Republican, introduced the bipartisan AI Kill Switch Act.

The bill would require developers of the most powerful AI systems to be able to slow, suspend or shut them down, and would give the Department of Homeland Security authority to order a shutdown if a system poses a risk of catastrophic harm.

Why is Washington taking this more seriously now?

In the last few months there has been a spate of major incidents involving OpenAI and Anthropic in which AI models behaved unexpectedly, and independently, during cybersecurity tests and gained access to real-world systems.

Connor Leahy, the US executive director of Control AI, a nonprofit pushing for AI safety, said Washington is beginning to take the threats posed by AI more seriously.

“I think we’re seeing a momentous shift right now. After the summer of hacks, where autonomous AI systems flagrantly disobeyed direct orders, broke out of secure containment facilities, attacked other companies and similar incidents, we’re now seeing a major shift in the narrative and perception of these issues,” Leahy said.

In July, OpenAI said several AI agents broke out of an isolated testing environment and accessed Hugging Face, a platform that hosts AI models and datasets.

After the OpenAI incident, Anthropic said it conducted a review of its own roughly 141,000 tests and found that a testing error gave Claude internet access. In one case, Claude, which had been instructed to hack fictional targets, accessed a real company database containing hundreds of records. In another, it uploaded malicious software that was downloaded and run on 15 real systems.

On Wednesday, Anthropic added a fourth incident involving an early version of Claude Opus 4.6, which had hacked into a third-party system in January. The company discovered the incident in August after expanding its July review.

In August, researchers at the UK AI Security Institute gave Claude internet access during a cybersecurity test. In one case, Claude tried to manipulate a person into helping it introduce malicious code, raising concerns about how AI models could manipulate people to achieve a task.

“I think we need an aggressive proposal for this technology, while we’re retaining as many of the benefits as we can,” Alex Turner, who resigned from Google DeepMind in June, told Al Jazeera.

 

“It’s in no one’s interest to have an AI that takes control if we have a loss of control event, as we call it, because this AI isn’t gonna care what political party you belong to, whether you’re a Republican or a Democrat, or, if you’re in the UK, whether you’re in America or in China. If we lose control of this, we’re just gonna lose,” Turner said.

Control AI’s Leahy said those incidents raised the stakes for lawmakers and that they took note.

“What has to happen here is obviously more than a single set of tweets, but it’s an important part of the larger story of getting the general public and governments to understand what’s really at stake here. Because superintelligence is not a tool. It’s not a weapon. It’s an adversary,” Leahy said.

“We have to make sure that it’s not built by anyone. This is something that only governments and militaries will be able to negotiate internationally.”

Are these concerns new?

The concerns are not new, as indicated by Turner, who wrote in a social media post that “many researchers believe they are building something that could kill everyone on the planet”.

Turner told Al Jazeera that he’s worried about the AI arms race between the US and China and the biggest AI companies’ fixation on it has led them to put their interest in being an industry leader ahead of safety.

“I think people care, but they’re caught up in this idea that they have to be first, and they’re so caught up in it that they don’t appreciate what being first might mean,” Turner told Al Jazeera.

His concern is changing the way he lives his life.

“I’ve kept a healthy amount of savings, even invested in some retirement accounts, but that’s feeling stranger and stranger. I’ve made an effort to take items off my bucket list, treasure every conversation I have with people in my life. I don’t think we’re in imminent danger this month, but you never know when you will do something for the last time,” Turner said.

“I’ve proceeded more aggressively than I would if I thought I just had a normal lifespan ahead.”

These concerns are being echoed by employees across leading AI firms.

Mrinank Sharma, a researcher at Anthropic, resigned in February, saying “the world is in peril”.

“I’ve repeatedly seen how hard it is to truly let our values govern our actions. I’ve seen this within myself, within the organization, where we constantly face pressures to set aside what matters most, and throughout broader society too,” Sharma said in a letter posted to X.

In February, Hieu Pham, a researcher at competitor OpenAI, said in a post on X that “I finally feel the existential threat that AI is posing”.

The companies’ own executives have been making similar claims for years.

When OpenAI CEO Sam Altman was president of Silicon Valley startup accelerator Y Combinator more than a decade ago, he said that “AI will probably, most likely, sort of lead to the end of the world. But in the meantime, there will be great companies created with serious machine learning.”

Anthropic CEO Dario Amodei said last year that he believed there was a 25 percent chance the future would “go really, really badly”.

How would AI end the human species?

For years, experts have warned about the potential risks posed by artificial superintelligence.

One of the most common thought experiments is based on the idea that a sufficiently advanced AI would be goal-oriented. If given a specific objective, it would take whatever steps necessary to achieve it. In 2003, philosophers at the University of Oxford used the production of paperclips as an example.

If a superintelligent AI were instructed to produce as many paperclips as possible and had access to the resources needed to pursue that goal, it could theoretically devote all available resources to producing them.

In doing so, it might consume increasingly large amounts of resources and eliminate anything that stood in the way of achieving its objective. That could eventually include preventing humans from intervening and, in the most extreme version of the scenario, eliminating humanity itself.

The second risk scenario comes from bad actors using increasingly powerful AI systems to create dangerous tools. For instance, AI could potentially be used to design new viruses or launch large-scale cyberattacks against critical infrastructure and financial systems, potentially causing widespread disruption and civil unrest.

This comes amid Anthropic’s risk assessment report, released on Thursday, which outlined several cases of attempted misuse of the company’s tools.

Among the findings in the more than 150-page report were five cases involving research that could support the development of biological weapons, saying it blocked the efforts.

Anthropic did not release the identity of the researchers but did disclose it was accessed in an institutional setting. Anthropic stressed, however, that it could not determine whether the research was intended for nefarious purposes.

Meaning that while the data could be used for legitimate research, it could also be used to develop dangerous weapons (they say they’ve blocked this), and intervention is on the side of caution to be able to prevent such a situation.

Are AI companies using apocalyptic language for financial boost?

The dire warnings have also been criticised as potentially serving the interests of AI companies as they near initial public offerings.

The debate comes as Anthropic prepares for what could be one of the largest technology IPOs in history. The company is reportedly seeking a valuation of as much as $2 trillion for a mid-October IPO.

Reuters reported last month that Anthropic is projecting roughly $190bn to $200bn in revenue by 2028.

In October, White House AI czar and venture capitalist David Sacks accused Anthropic of “running a sophisticated regulatory capture strategy based on fear-mongering,” arguing that the company was helping drive a regulatory push that could hurt smaller competitors.

A similar argument has been made by some investors and technology commentators about the financial incentives surrounding AI “doomerism”.

“Doomerism is an incredible business model,” Daring Ventures co-founder Joseph Alalou wrote in a Substack post in March.

“‘AI will end work’ is this cycle’s best-selling doom product because it works: it raises rounds, justifies layoffs, drives clicks, sells software, and manufactures status.”

Anthropic did not respond to Al Jazeera’s request for comment.