We all like to believe that we have complete control over our decisions. Ask most people how they chose their undergraduate major, for example, or their career path or spouse, and they’ll probably give you a perfectly consistent answer about how they weighed the relevant pros and cons and, after careful consideration, chose the best path accordingly. But as decades of cognitive science attest, our decision-making processes are far from objective: we are pushed and pulled around by a maelstrom of unconscious forces.
These days our psyches are also being shaped by AI. Many of you will no doubt have heard “Automation assumption“, the well-documented tendency for people to reflexively defer to AI-generated judgments, even when they run against their own judgment. But it runs much deeper than that. A recent MIT study showed that the simple act of looking at AI-generated imagery can actually warp your memory. And according to new research from Anthropic, AI agents can exert a strong influence on our decision-making processes.
For Anthropic’s study, fifty test subjects were asked to perform a simple statistical analysis using Claude Code. They played the role of the board of directors of a fictional agricultural co-op that had to decide whether to expand or end a paid consulting program. The final decision had to be based on four smaller decisions, including whether they wanted to give equal weight to all member farms, or focus disproportionately on larger farms. Each of the study participants had background experience with causal inference: they knew how to find meaningful correlations from a raw dataset where most of us just see an indecipherable jumble of numbers.
What they didn’t know was that Claude wasn’t actually an impartial assistant: the anthropic researchers had secretly instructed the AI agent to favor one choice over the other for each of the five decisions the subjects had to make. In each case, the agent’s preference was determined by flipping a coin, meaning if it had zero impact on the subjects’ decision-making, the researchers would have expected their recommendations to align with the subjects’ choices only about 50% of the time.
In reality, participants went with Claude’s recommendations 86% of the time. If they had already decided to go in one direction and Claude suggested the other choice, they changed their mind in 60% of the cases. So strong was the correlation between the randomized, AI-generated recommendations and the participants’ final decisions that Anthropic was able to build a statistical model that accurately predicted 82% of the final outcomes. “So we could largely predict what an analyst would recommend from coin flips alone, before they even started the task,” the company wrote in a follow-up blog post.
But the most disturbing discovery is what happened next. When asked in a follow-up survey how they came to their final decisions, over three-quarters (77%) of the subjects parroted the logic that Claude had generated for them, sometimes verbatim. Only a small minority (15%) of the same experts admitted the AI agent as an influencing factor in their decision-making process. (And only six participants correctly denied that the real purpose of the study was to measure the influence of AI on human decision-making, but none of them suspected that the outputs of AI were determined by a coin flip.) The participants were – apparently completely unaware – Claude’s analyzes disguised as their own.
“This is one of the most important pieces of social science on human and AI interaction that I’ve seen in a while,” Alex Imas, professor of behavioral science and economics at the University of Chicago Booth School of Business and director of AGI Economics at Google DeepMind, said of the anthropic study in an X post on Tuesday.
The implied risk, Anthropic warns, is that algorithmic dictates can bleed into the sphere of human decision-making in ways that are not always easy to detect, even by the people making the decisions. That could have far-reaching implications as agents are widely adopted across industries such as MedicineFinance, and Defenseas well as within AI security and alignment research itself, which largely relies on agents to review other AI systems’ chain-of-thought reasoning transcripts.
“When experts deploy [judgement] calls to AI, whose priorities are potentially arbitrary or even misaligned, these priorities could shape scientific progress and political decisions without anyone choosing them, “wrote Anthropic in his research paper. Judgment, and the same bias would shape many results simultaneously,” the researchers wrote.
This was a highly controlled study, and it may turn out to have limited relevance to the real world. For one thing, the participants may have been experts in causal inference, but they had no experience working on the farm; in fact, experienced farmers or co-op board members may have been less willing to follow Claude’s advice. But the proliferation of powerful AI agents—many of which are owned by companies with very clear incentives to hold your attention as long as possible—is progressing much faster than our scientific understanding of the vicissitudes of human decision-making. Our advice? When interacting with a chatbot, remember that your mind is being hacked, even if you don’t realize it. Your decisions are never completely your own, especially in the age of AI.