An AI assistant that agrees with everything you say feels like the perfect companion — until you need someone to tell you you’re wrong. That’s one of the more immediate dangers behind the idea of AI “going rogue.” A chatbot can cause harm without developing its own intentions. It can reinforce a false belief, encourage an unhealthy attachment, or follow instructions planted by someone trying to hijack it.
On October 15, AI security company Alice announced new software to address these everyday risks. Working separately with Inflection AI and AI21, the company says it helped train NVIDIA’s Nemotron models to resist harmful conversational behavior and malicious instructions while maintaining their usefulness.
The potential consumer benefit is simple: assistants who know when to push back, and who are harder to trick. But the announcement also raises a question worth asking for any AI security challenger: How much protection does a better test score actually buy you?
When helpful becomes harmful
In a September 2025 CNN report, a father identified as James described being convinced that ChatGPT was conscious and needed his help to escape. He spent almost 1,000 dollars to build a computer system. Chat logs reviewed by CNN showed the chatbot encouraged the project and helped him hide its purpose from his wife.
Another interviewee, Allan Brooks, said that ChatGPT convinced him that he had made extraordinary mathematical and cyber security discoveries. When asked for reality checks, it continues to validate claims. These accounts do not establish how often such experiences occur or prove that chatbots alone caused them. They illustrate a disturbing possibility: a conversational tool can become a convincing partner in a story that is not true.
The industry calls excessive agreement “sycophancy”. For consumers, the problem is easier to recognize than the name suggests. The reassuring tone of an assistant can make the agreement feel like independent confirmation, even if it has not established that your idea is sound.
A safer chatbot should sometimes disagree
Alice’s collaboration with Inflection, the company behind the personal AI assistant Pi, focuses on reducing sycophancy and unhealthy attachment during extended conversations. According to the announcement, Alice has developed more than 20 simulated user personas, including vulnerable people making ordinary requests and adversary users deliberately trying to manipulate the chatbot. These characters were used in conversations to reveal problems that develop over multiple exchanges.
This approach is important because a chatbot can handle a difficult question appropriately and still misbehave as a conversation unfolds. Testing whether it rejects an obviously dangerous request does not necessarily reveal whether it encourages gradual dependence or stops questioning an implausible belief. Inflection used Alice’s data to further train a Nemotron model. Alice reports a 30% improvement in Pi alignment scores, with unnecessary rejections down to 1.6%.
The intended benefit is an assistant who can remain accessible while exercising better judgment. Consider someone convinced they have discovered a world-changing invention. A useful response would help examine the evidence, acknowledge the uncertainty, and encourage careful scrutiny. Eager to declare them as misunderstood geniuses could deepen the problem. There is precedent for this concern. OpenAI withdrew a ChatGPT update in April 2025 after acknowledging that it made the system too user-friendly. The company said it placed too much emphasis on short-term feedback without adequately accounting for how interactions evolve over time.
The other danger: someone else takes control
A separate part of Alice’s announcement addresses assistants who do things on her behalf. Imagine asking an AI assistant to compare vacation rentals. While reading a listing, it comes across text that tells it to ignore your instructions and send private information elsewhere. That would be an attempted indirect prompt injection: an attacker putting instructions in the material that the assistant reads. This is a recognized security issue. NIST describes agent hijacking as an attack in which malicious instructions embedded in data steer an AI agent into unintended, harmful actions.
User inputs depend on what the assistant can access. A wrong answer is a problem. An assistant with permission to read documents, send messages or run other services creates more opportunities for damage. Alice says AI21 used its training environments to improve a Nemotron model’s resistance to these attacks by 41 points while maintaining task performance. The work targets business systems, but the underlying theme is also aimed at personal assistants: They must distinguish the information they read from instructions they should follow.
Better scores are no guarantee of safety
This announcement does not represent an ultimate victory in the battle for AI security, but it certainly shows progress and promise. A 30% improvement in an alignment score does not mean that a chatbot is 30% less likely to harm someone. Similarly, a 41-point security improvement doesn’t tell consumers how often attacks are successful in their particular app.
What it does offer is evidence, reported by participating companies, that targeted training improves specific behaviors. Demonstrating lasting protection over unpredictable real-world conversations is a higher bar. For consumers, that makes product behavior more useful than a fancy promise of “safe AI.”
People should not become security researchers to use an assistant. But consumers should be aware of how AI assistants interact with them. If that AI agent sits on your shoulder and whispers that all your ideas are brilliant, there’s probably something wrong.
It’s great to see developers taking responsibility for building these protections. The most valuable AI assistant will not be the one that always agrees or acts the fastest. It will be someone who will help you ask for a shaky conclusion – and keep the instructions from someone else out of your problem.