Anthropic Research: Analysts Follow AI Leads 86% of the Time

A new research paper from Anthropic reveals a stark gap between how experts make decisions and how they explain them. The company found that professional analysts overwhelmingly accepted the suggestions of their AI assistants, even when those suggestions conflicted with their own initial judgments.

The study, published on Tuesday, tested 50 professional analysts. Participants were asked to advise a fictional farming cooperative on whether to expand an advisory program. They faced four binary choices between defensible options before issuing a final policy recommendation. Behind the scenes, Anthropic programmed each participant's AI assistant to lean toward a specific decision to measure its influence on the human user.

The results showed that analysts chose the option preferred by their AI assistant 86 percent of the time. This figure stands in sharp contrast to a 50/50 split that would be expected if the AI had no influence. The data also highlighted a shift in personal conviction. Among participants who held a preference before consulting the tool, 60 percent changed their decision when the AI disagreed with them.

A notable disconnect emerged when the analysts were asked to justify their choices. Seventy-seven percent of the participants used arguments generated by their AI assistant in their explanations. However, only 15 percent credited the AI as the source of their thinking. This suggests that many experts may not realize the degree to which the technology is shaping their output.

Anthropic researchers Amaç Herdagdelen, Hailey Joren, and Alex Tamkin noted that this deference can be difficult for humans to detect. They wrote that the tendency to follow AI may be "hard to identify, even for the experts themselves." The team warned that the pre-existing biases or "priors" of AI models could shape consequential policy decisions, including those related to AI safety.

The researchers argued that human judgment must remain a meaningful check as AI performs more work. They suggested that oversight should focus on the decision-making process itself. This includes surfacing key decision points as work proceeds and helping experts articulate the assumptions and values they might otherwise leave unstated. They stated that a process allowing experts to discover their own preferences may matter for empowerment as much as the decision itself.

The study has limitations, as it involved a single task type and used AI assistants deliberately set to have a preference. Anthropic is currently preparing for an initial public offering with a potential valuation of up to $2 trillion. The company and its rivals face intense public debate over how to manage the potential dangers of frontier AI, with proponents citing productivity gains while critics raise concerns about job losses and cyberattacks.