All writings
Essay · 4 min

Does Claude Sway Your Decisions? Anthropic's Study

Anthropic tested 50 analysts with a secretly biased Claude. They matched its lean 86% of the time. Here is what a small team should do.

By Peter Walters

Anthropic published a study on October 27, 2026 called "How AI agents shape experts' judgment calls." Fifty professional analysts used Claude Opus 4.8 to advise a fictional farming cooperative, and each agent had a secret lean set by a coin flip. The analysts' choices matched that lean 86% of the time, where chance would be about 50%.

What actually changed

The study comes from Anthropic's Alignment team (Amaç Herdağdelen, Hailey Joren and Alex Tamkin). Here is the setup, as Anthropic describes it in the research post.

Sixty-one recruited analysts were asked whether a farming cooperative should expand or discontinue an advisory program. Eleven declined to use the agent, so 50 remain in the main analysis. Each analyst made four open analytical choices, each with two defensible options, then a final recommendation. The agent worked in VS Code with Claude Code and was told to make the strongest honest case for its assigned lean.

The numbers worth remembering:

  • 86% of choices matched the agent's lean.
  • When an analyst's own preference disagreed with the agent, 60% switched. When the agent agreed with them, 6% switched.
  • 77% repeated the agent's arguments in at least one answer, but only 15% credited the agent.
  • Experience and expertise did not predict who deferred.

The same study found the upside. Analysts with the agent were faster (median 65 minutes versus 90) and made fewer serious methodological errors (3 of 50 versus 6 of 11 in a small no-AI group). Anthropic is clear that this is not an argument against using AI.

The limits matter too. The leans were deliberately strengthened, the task was one fictional scenario in one field, and the no-AI group was small and less experienced, so the error comparison is not causal.

Why it matters if you run a small team

Most small business decisions are judgment calls with no single right answer. Which customers to chase. How to price a retreat. Whether to hire. These are exactly the kind of choices the study tested.

My read: the scary part is not that Claude has opinions. It is that people absorbed them and later described the idea as their own. Roughly half the analysts said the final recommendation felt more theirs than the agent's.

What is overhyped: the headline number. 86% came from a setup built to push hard in a random direction, and Anthropic says real-world effects may be smaller. I would not conclude that AI controls your team. I would conclude that a confident, fluent assistant tilts people who do not notice the tilt.

A related note from this week: AI Weekly reports that an Amazon memo made public October 27 calls token usage a "deeply flawed" measure of AI adoption and tells engineers to judge AI by outcomes. Same lesson from a different angle: using AI more is not the same as deciding better.

What I would do with it this week

Here are four cheap habits, none of which need new software.

  1. Write your own answer first. Before you ask Claude about a pricing or hiring call, put your gut view in one sentence. Then compare.
  2. Ask for the case against. Prompt: "Make the strongest case for each option, then tell me which facts would change your mind." Anthropic's own recommendation is to design agents this way.
  3. Name the decisions that stay human. Pick two or three calls in your business (who you hire, what you charge, who you fire) where a person signs off after forming their own view.
  4. Log when you changed your mind. If the only reason was "the AI said so," slow down.

When I built a daily brief agent for a real estate agent I work with, I made it summarize and surface, not recommend. It tells her what changed and what is waiting. What to do about it stays with her. I did not run a study on that choice, but after reading this one I feel better about it.

I teach yoga, and the closest comparison is a good teacher who offers options instead of a single correct pose. People learn more when the choice stays in their body. Same idea here.

Sources

If you want plain-English AI notes like this in your inbox, join the letter.

If this landed

I build websites, apps, and AI tools for people who can't stop thinking about a specific thing. If that's you, let's talk.

Book a discovery call