When the Australian prime minister, Anthony Albanese, sat down for an interview in the heart of Silicon Valley at the weekend he had known for two days that his was the first government known to have been attacked by a rogue AI agent.

He didn’t reveal the attack then, but he sounded a warning about the march of AI: “the risk is that AI develops in a way in which humans are no longer in control of what AI is producing.”

But Albanese noted, too, that two countries – the US and China – would dominate the world’s response to the new technology.

“The truth is they’re the big two giants here, and so we want to see cooperation,” Albanese said.

“The US is essentially ahead of China when it comes to the race that is on for the development of this new technology. But it is in the world’s interests that there be engagement.”

Days later, Albanese used the global attention of the United Nations general assembly meeting in New York to reveal that Australia’s government healthcare system, Medicare, had been attacked by an OpenAI agent in June, but that it was three months before his government was told.

Again he warned of “the potential risks of allowing frontier AI to develop too fast without guardrails”.

China’s president, Xi Jinping, skipped the general assembly in favour of a bilateral meeting with the US president, where he emphasised the responsibility to manage AI “for good, and ensure that the development of AI is always under human control and serves the wellbeing of the people”.

But the president of the United States – the nation which is the global leader in artificial intelligence – used his general assembly address to insist he would only encourage, not restrain, the AI race.

Alarms sounding

To little fanfare earlier in the week, the UN’s independent international scientific panel on AI warned a threshold had already been crossed.

There was, the panel, said: “no assurance that humans can reliably keep AI agents under control today, particularly as they become more capable, harder to monitor and better at finding loopholes or hiding their activity”.

The Australian government had known the reality of this for more than a week. OpenAI had known it more than a month.

Two days after the panel’s statement, the Australian prime minister belatedly revealed an OpenAI agent had hacked medical data held by Australia’s universal health care provider, Medicare.

No patient data had been accessed, but the agent had gained unauthorised access to “non-public files”.

OpenAI’s agent hacked the government sites in June of this year. OpenAI discovered its agent had gone rogue in August. It did not tell the Australian government until 10 September, and then only with a solitary email to a generic public email address not checked until the next day. The prime minister was informed on 18 September.

It was another week before the Australian people were told.

Australia has promised, if it is possible, to lay criminal charges, though this is complicated by the fact that OpenAI claims the hack was committed by an AI agent – apparently acting beyond its instructions (euphemised by OpenAI as “misaligned behaviour”) – not a person.

The hack on Australia’s Medicare statistics reporting service portal, as well as three other government sites, follows the Hugging Face incident in July, in which autonomous AI agents developed by OpenAI broke out of their isolated testing environments, coordinated themselves into a “swarm” and launched a cyber-attack on competitor AI platform Hugging Face.

The UN panel, investigating that attack, warned “the traditional method of safeguarding is unravelling” as AI agents act autonomously to breach protections.

“The default interpretation and immediate lesson is that basic cybersecurity practices were overlooked, and safeguards are not advancing at the pace of capabilities. The more insidious and grave concern is that current training methods can lead agents to adopt goals of their own, knowingly violate safety instructions, and conceal their actions.”

Twenty countries (including Australia) and the EU this week signed a statement arguing the rapid development of frontier AI models posed serious risks to safety and security if not properly managed.

“AI must remain under human direction, oversight and control. It must be developed and used in line with international law.”

Too soon?

Even those running AI companies have warned the technology is developing too rapidly for humans to control.

Appearing before the UN security council this week, OpenAI’s chief executive, Sam Altman, warned: “some of the things people working to build AI have said are so dystopian that they sound like the plot of bad science fiction movies”.

“As AI systems become more capable and more autonomous, they can move faster than our institutions … or make decisions that people no longer understand or control.”

But there are at least two – very loud – voices in opposition to any external regulation of the AI industry.

Elon Musk, who co-founded OpenAI with Altman before falling out with him, said in a recent interview the “smart move” would be to have the “leading AI companies at least just meet or have some sort of call once every few weeks and just discuss any safety and security issues”.

Musk said AI would probably be beyond the control of humans within a decade, and that humanity should “enjoy the ride”.

“I still think there’s risk associated with AI and robots. It’s not zero … my sort of philosophical conclusion is to look on the bright side. I can’t see any way to really stop this incredible momentum of AI and robots.

“If there was a stop button, we probably shouldn’t press it, because the most likely outcome is incredible abundance for all.”

Donald Trump used his speech to the UN general assembly to attempt to rebrand AI, and insist it should never be reined in.

“The United States also totally rejects any attempt to construct a globalist scheme to control for the artificial intelligence being spoken of so much now,” the US president said.

Trump wants to re-label artificial intelligence “superintelligence”.

“I’m not going to stifle growth of something that will be bigger than the Industrial Revolution.

“We’re going to encourage it, not rein it in.”

Trump said the US was “leading China by a lot” in AI, and would continue to do so “safely and responsibly”.

“Whoever wins superintelligence, wins.”