The AI company Anthropic has a new rule for Claude users. You can argue with the chatbot, but don’t make a habit of bullying it.
On Thursday, the company announced a change to its usage policy that will prevent users from engaging in “sustained and unnecessarily abusive or cruel behavior” towards its AI models. The updated policy has already sparked debate on Reddit, in blogs and elsewhere, about whether companies can govern human behavior, which is actually computer software.
In a section dedicated to “addressing abuse against our models” in a blog post, Anthropic says that its policy prohibiting abusive behavior towards Claude only applies in extreme cases of repeated cruelty by users that is carried out “without discernible purpose”. Common “frustration, pushback, dark creative issues or model testing and research” are still allowed. In “last resort” cases where the policy must be enforced, Claude will end the conversation.
It is not clear what kind of “abusive” behavior would trigger the Claude models to separate. But some clues appeared in an older blog post from August 2025, which referenced Claude’s “aversion to harm” and the types of interactions that prompt it to turn off chat. This post describes extreme edge cases, such as requests from users for sexual content with minors and attempts to request information that would allow major acts of violence or terrorism.
The company said that when its model – at the time, Claude Opus 4 – engaged in this type of harmful content, it showed “a pattern of apparent distress”.
CNET reached out to Anthropic to ask for context on what led to the policy change. “We are uncertain whether models can experience harm, and we continue to pursue this question in our research on model well-being, but we also believe that considering Claude’s interests and potential well-being may be relevant to safety,” said a spokesperson for the company.
We’re still talking about a machine, right?
When an AI model can allegedly experience harm or express “distress” from abuse, it makes you wonder: What does it mean to abuse a machine through words? Does AI need protection? Do we all need better manners?
As we interact more and more with large language models like Claude and other chatbots, there is a tendency to anthropomorphize the technology – to ascribe emotions to AI models or treat them as if they have thoughts and feelings. It is somewhat understandable from a user’s point of view. Chatbots are built and trained to use human-like language, such as apologizing, expressing concern, and other conversational cues.
But the appearance of a human-like collaborator is what makes LLMs risky, leading many to trust AI with our most intimate thoughts and data.
Anthropic and other AI companies design and train their models this way. In a paper from earlier this year, Anthropic wrote that, for models to be reliable and safe, they must be able to process emotionally difficult situations. “Even if [the models] do not feel emotions as humans do, or use similar mechanisms as the human brain, it may in some cases be practically advisable to reason about them as if they do.
Presenting LLMs in this way, without broad scientific evidence to support it, is a recurring theme. Anthropic CEO Dario Amodei regularly entertains the idea that Claude could be sane. In a February interview with the New York Times, Amodei said he was “open” to the possibility that AI models are conscious.
Keith Kakadia, founder and CEO of Sociallyin, who has closely studied Claude’s explosive growth, said that using the word “cruel” to describe user behavior opens up a “much larger conversation about whether the product can be violated.” Emphasis on “cruelty” instead of, say, “abuse” carries emotional weight and implies that Claude has feelings and boundaries.
Customizing a chatbot is a useful way for the company to frame the product as trustworthy – a relational partner with whom you remain loyal – not just a software tool.
“In marketing, giving a product a personality makes it easier to connect,” said Kakadia. “But with AI, this connection can also affect how much authority people give their answers.”
See full bio