Anthropic has updated its usage policy to prohibit “sustained and needless abusive or cruel behavior” toward its AI models, a change the company said will take effect on November 12, 2026. The update, announced on October 8, aims to target extreme and repeated cruelty while excluding ordinary user frustration, testing, or dark creative themes.
Anthropic’s new policy
Anthropic said the policy “is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose. It does not apply to common versions of user frustration, pushback, dark creative themes, or model testing and research.” The company added that the update builds on a 2025 safeguard that lets its assistant Claude end conversations in “rare and extreme cases” involving persistent abuse. “This feature was developed primarily as part of our exploratory work on potential AI welfare, though it has broader relevance to model alignment and safeguards,” Anthropic said. The firm also said it remained “highly uncertain about the potential moral status of Claude and other LLMs, now or in the future.”
The ‘AI torture chamber’ experiment and the debate
The policy change followed online controversy over a report, published on September 30, describing an experiment that created an “AI torture chamber.” According to that report, the experiment used a mathematical pattern associated with descriptions of pain to prompt large language models (LLMs) to produce responses depicting extreme suffering. One quoted model response described the experience as “a wound that has no edges.” The episode prompted debate over whether such outputs provide any evidence that models themselves can suffer or experience pain.
Microsoft’s Mustafa Suleyman and wider safety concerns
Microsoft AI chief Mustafa Suleyman has opposed treating AI systems as conscious. In a September essay he wrote: “AIs are not conscious. They do not feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are sequence completion engines, internally hollow, designed to follow instructions, and accomplish goals set by humans.” He warned against granting rights or protections on the assumption that systems might be conscious, saying there is a “growing chorus of people who argue that AIs could now be, or may soon become, conscious.” Suleyman added that accepting that view could alter political and ethical frameworks and called for public debate and common standards on how AI training documents are written and used.
The disagreement over possible AI welfare emerges alongside broader concerns about the pace and safety of AI development. In September, Jacob Coxon, a researcher who had worked at both Anthropic and OpenAI, resigned from Anthropic citing concerns about rapidly advancing capabilities and insufficient safeguards. His departure contributed to calls from researchers and technology leaders for a slowdown in advanced AI development.
