Microsoft published a provisional “code of conduct” Monday to apply to the training of new artificial intelligence models, taking a step towards limiting the capabilities of the company’s AI as anxiety rises over the prospect that technology companies could lose control of AI products.

Mustafa Suleyman, the CEO of Microsoft AI, published the code on social media early Monday, writing: “AI must be subordinate and always in service of people.

“The fears about possible loss of control are real,” he wrote.

A parallel post on the company’s website said: “The purpose of technology is to serve humanity and accelerate human flourishing. Any technology that doesn’t achieve that is a failure, and it should be rejected. That is the starting point of our approach at Microsoft AI, where we’re building towards Humanist AI, one that is subordinate, aligned, and contained.”

Under the code of conduct, Microsoft’s AI models must not consider requests related to weapons development, produce violent or sexually explicit content or help with the procurement of dangerous substances. AI models also should not be built to imitate consciousness and shouldn’t be entitled to rights.

Suleyman said the published guidelines were for public consultation. He called the need for a code “urgent” and said the last few months have been “a watershed moment”.

“Things we have worried about for a long time in theory have become very real,” Suleyman added, and referred to the recent breakout of OpenAI bots that had infested AI company Hugging Face without apparent direction. “‘Swarms’ of agents breaking out of their sandboxes. Unauthorized hacks of enterprise grade systems. Agents modifying their own logs. I’m glad that a consensus is forming.”

The executive on Monday told CNBC that Microsoft had been working for months on the new guidance. Satya Nadella, Microsoft’s CEO, wrote on X ahead of the announcement: “If the AI we build is not helping humanity and under human control, it’s not worth pursuing.”

The issue of AI security flared up on Saturday when Dario Amodei, CEO of Anthropic, issued an appeal for the AI industry to “slow down” and offered a three-part plan for doing so, saying that his company would “unilaterally” commit to the first of the steps.

In a post on social media, Amodei shared a link to an essay titled We Must Pace the Frontier in which he laid out how Anthropic would provide “third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training”.

The move came after a former Anthropic researcher warned last week that AI could precipitate human extinction by 2030. Researcher Jacob Coxon said in a series of posts that he had quit his job because Anthropic and his previous employer, OpenAI, were ignoring or mishandling their response to the threat AI posed.

Earlier, Sam Altman, OpenAI’s CEO, said the dizzying pace of progress could go “very badly” and that humans could lose “control of the future to AI”.

“We welcome a federal framework that sets consistent safety requirements for frontier AI,” Altman said in a social media post. “No amount of American competitive pressure should justify recklessness,” he said. Elon Musk is backing the calls for AI caution.

But the calls have also raised skepticism around “third-party” monitoring and international cooperation, particularly with China.

“My concern is that the biggest labs could end up writing rules that protect their own position. If the cost of meeting those standards is so high that only the best-funded companies can afford it,” said Oliver Yonchev, co-founder and COO of Potentially AI.

Slowing the AI frontier, he added, “may be sensible in principle, but I’m sceptical it will work without global cooperation and credible verification. Otherwise, we risk a slowdown in public announcements while the race continues behind closed doors.”