Algorithmic Aversion

The model misses one forecast, and the miss is easy to see. It is a number, a name, a route, a diagnosis that was wrong in a way you can point at. The colleague who misses more often, more quietly, suddenly feels like the careful option. You have watched the machine fail. You have not kept the same tally for the person.

Dietvorst, Simmons, and Massey named this algorithm aversion. In forecasting tasks, people were less willing to tie their payoffs to an algorithm after they had seen it work, even when they had seen it beat the human alternative. Confidence fell faster for the algorithm than for a person after the same kind of mistake. A human error was absorbed as "someone being wrong." An algorithmic error was absorbed as "the system is not to be trusted."

The studies were not a claim that every model is better than every person. They used tasks where the evidence-based forecast had the stronger record, and people still walked away from it once its imperfections were on display. Perfection was the standard for the machine. A looser standard was available for the human, including for the participant's own guess.

There is a matching over-trust on this site, Automation Complacency: the green check, so you stop looking. Aversion is the swing after a miss. Both skip the record. One skips it because the system looks finished. The other skips it because the system looked fallible once.

Refusing a model can be the right call. If it was trained on the wrong cases, aimed at the wrong goal, or opaque in a domain where you must be able to explain the decision, a person may be the better tool. The bug is letting one seen error outweigh a better average, then calling that caution.

• "It was wrong once. I'll do it myself from here."
• "I'd rather have a person. At least a person can be accountable."
• "The model missed that case, so the whole approach is junk."
• "My estimate feels safer. I understand how I got it."
• "We saw the chatbot invent a citation. We shouldn't use any of these tools."
• "The human forecast is worse on paper, but I trust it more after that miss."
• "If I can't tweak it, I don't want it making the call."

The forecast: a statistical model has been better than the sales team for a year. One quarter it misses a spike everyone can remember. The next planning meeting goes back to the loudest person's number. The year of smaller misses by the team is harder to picture than the model's one famous error.

The schedule: routing software sends a driver into a closed street. The driver is right to override today. Next week the team turns the software off and returns to a whiteboard that, on average, produced longer days. One street became a policy.

The draft: a writing tool mangles a name. You catch it. From then on you won't let it near a first pass, including for chores where your own blank page is slower and no more accurate. The mangled name is vivid. The blank page's ordinary errors are not in the comparison.

The screen: a diagnostic aid flags a result that turns out benign, or misses one that a clinician catches. The aid's overall rate never gets pulled up beside the clinician's overall rate. The story that remains is the case you can tell at dinner.

Decisions get worse in the specific way the experiments measured. People left a more accurate forecast and tied their outcomes to a less accurate human one, including themselves. The loss is not philosophical. It is the extra error you accepted in order to avoid depending on a tool you had seen fail.

Teams also bounce. A tool is adopted with little checking, then banned after a visible miss, then quietly reused with no record either way. Neither swing is a method. The people who needed a stable division of labor, what the model does, what a person must review, never get one.

There is a control piece that matters. In follow-up studies, people were more willing to use an imperfect algorithm when they could adjust its forecast, even by a little, and they performed better when they did. The aversion is partly about being stuck with an error you cannot touch. That is useful to know, and it is not a reason to "correct" a good model into a worse one just so it feels yours.

Human error comes with a story: a bad day, a missing fact, a judgment call. Algorithmic error comes without that story, so it reads as a property of the system. Once you code the system as broken, further use feels naive. A person's next guess still feels like a fresh try.

You also see the machine's mistake in clean units: a score, a wrong street, a fabricated source. Your own mistakes are smeared across ordinary days and easy to forget. Comparison by memory favors the human. Comparison by record often does not.

Control matters. Dietvorst, Simmons, and Massey found that the preference for a modifiable algorithm was about having some control over the outcome, not about needing a large override. People wanted a hand on the result. They did not need a large hand for the willingness to return.

Dietvorst, Simmons, and Massey's 2015 paper in the Journal of Experimental Psychology: General reported five studies. Participants decided whether to tie incentives to an algorithm's forecasts or a human's. Seeing the algorithm perform made them less confident in it and less likely to choose it over an inferior human forecaster. That held even for people who had seen the algorithm outperform the human. The authors' account is the uneven drop in confidence: the same kind of mistake costs the algorithm more trust than it costs a person.

Their later Management Science studies, first published online in 2016, asked how to loosen that refusal. Participants were more likely to use an imperfect algorithm built by experts when they could modify its forecasts, including when the modification they were allowed was severely limited. They also did better. Feeling some control increased satisfaction and later willingness to use an algorithm again. The papers are about forecasting under incentives. They are not a blanket ranking of humans and machines in medicine, law, or art.

• One error is enough for you to retire the tool, and you have not counted your own errors on the same task.
• You describe the model as "wrong" and the person as "having an off day."
• You prefer a method you can narrate over a method with the better record.
• The ban arrived immediately after a story you can retell, not after a review of rates.
• You would accept the model's help if you were allowed to adjust it, and you have not asked whether the adjustment helps.

Keep the right to override. Lose the right to treat one override as a personality test of the entire tool.

• Before you abandon a model, write two rates: its misses and the human alternative's misses, on the same task, over more than the case you remember.
• Separate "wrong for this case, override it" from "worse on average, stop using it." Those are different sentences.
• If control is the real issue, decide in advance what a person may adjust and what they may not. Small, named adjustments beat a vibe-based rewrite.
• Log the override. A month of logs will show whether the human corrections are helping or reintroducing the errors the model was there to reduce.
• Do not flip from aversion into blind use. A tool that is unfit, opaque where explanations are required, or pointed at the wrong goal should stay unused for that job.

The model was wrong. Why shouldn't I stop trusting it?

Stop trusting it for the case it got wrong. Stopping for every later case is the aversion. Ask whether the human process you are about to substitute has a better record or only a more forgivable texture.

What if I need to understand the reason, not just the score?

Then a black box may be the wrong instrument, and preferring a person is a requirement of the job, not this bug. Algorithmic aversion is refusing a better record because imperfection was visible. A demand for an explanation you can defend is a different demand. Name which one you are in.

Isn't it good that I can modify the output?

Sometimes. The follow-up studies suggest a little control brings people back to a useful model, and that a large override is not what their willingness depended on. Modify when you have information the model lacked. Do not modify to soothe the discomfort of letting an imperfect score stand.

Judge the miss against the alternative's misses. Keep a human hand where the hand has something to add.

The famous miss

The mangled name

The override

Practice this pattern in the Reframing App - capture the trigger, label it (like Algorithmic Aversion), check evidence, and write a more balanced thought.