Technology experts have warned revelations an artificial intelligence agent hacked Medicare’s internal systems will not be the only dangerous breach of government data and have called for Australia to boost its protections against the growing risk.

The prime minister, Anthony Albanese, challenged the OpenAI boss, Sam Altman, on Thursday after the company’s agent infiltrated systems run by the Australian Institute of Health and Welfare, Victoria’s Department of Health, the New South Wales Bureau of Crime Statistics and Research, and the Medicare statistics reporting service portal of Services Australia.

OpenAI alerted the government earlier this month to the June hacking via an email to a public-facing address, a situation Albanese called “obviously unacceptable”.

But the Australian Council on AI Strategy chief executive, Anna-Maria Arabia, said the case was unlikely to be an isolated incident.

“All of the evidence shows that our operating systems are vulnerable,” she told Guardian Australia.

“Frontier AI now has capability to expose those vulnerabilities at a rate quicker than we can keep up, quicker than we can patch them.

“When the companies are undertaking tests in what they think are secure environments, and when there are breaches of those environments and these incidences do happen, whether it’s accidental or not, what we’re seeing is the frontier AI capability exposing these vulnerabilities.

“All evidence suggests that there is more of this to come.”

Arabia said Australia needed to quickly enhance capability to detect and report incidents, and the country should host AI training labs here.

Johanna Weaver, Australia’s former chief cyber negotiator at the United Nations, agreed more incidents were inevitable.

Weaver is a member of the advisory board to the minister for government services, Katy Gallagher, and the executive director of the Tech Policy Design Institute.

“Cybersecurity experts have been warning that frontier models and AI agents could expose vulnerabilities in critical systems. What we are seeing now is the tip of the iceberg.

“Governments need to draw a clear line: if companies cannot control their AI systems, they should not release them publicly.”

The US Studies Centre expert Olivia Shen warned AI companies should not be allowed to decide on their own disclosure obligations for hacks and breaches.

“We just don’t know how big the problem is. It could be the tip of the iceberg, but either way, we can’t be ignoring the risk.

“It’s all happening at a time when Australia is designing our national standards on AI. It hasn’t been entirely clear if those national standards were going to be very hyper-focused on datacentres and leave governance questions as a bit of a bolt-on.

“I think this strengthens the argument that you need to have some pretty clear standards, even just based on mandatory incident reporting, built in.”

Intelligence agency the Australian Signals Directorate (ASD) is reviewing how prepared the government is to block and respond to hacking by AI. Officials will look at policies around how AI companies should report cyber-incidents to the government, and how cooperative companies should be during and after an attack.

It will also investigate whether the current laws and systems are adequate to stop AI, and how government systems can be strengthened.

The shadow industry minister, Andrew Hastie, called for Australia to develop its own domestic AI capability, instead of relying on the US.

“If there’s rogue AI agents out there, we need to have our own defensive AI agents protecting Australian government data, our private sector, and other things that are important to us,” he said.

The Greens demanded Labor call in the new US ambassador, David Brat, to establish what President Donald Trump knew about the attack.

“This breach by a foreign AI company on an Australian government database is deeply alarming and brings home the risks that these out-of-control tech corporations pose,” acting leader, Mehreen Faruqi, said.

“The fact that the government did not even know it happened is disturbing.”