AI Advice Needs a Calibration Gate, Not Just an Accuracy Score

An AI feature can give a user a wrong answer and still make that user more confident in the wrong answer. That is the product risk most model evaluations miss.

For a TPM, this changes the launch question. You are not only asking whether the model produces a correct answer often enough. You are asking what happens to the human decision after the answer appears. Does the user verify it, challenge it, or stop thinking because the system sounded certain?

Two preregistered experimental programs published in 2026 point to the same failure pattern. Wrong AI advice can reduce a person's willingness to say "I don't know," lower accuracy, and raise confidence at the same time. The studies used online participants and deliberately difficult tasks, so they are not a universal estimate for every product. They are a useful stress test for a class of risk that belongs in every AI launch review.

The practical response is a calibration gate. It sits beside model evaluation, not underneath it.

The threshold that matters

In one set of five experiments, Marcoccia, Quattrociocchi, and Capraro tested whether people would suspend judgment when they did not know an answer. The program included 3,132 participants across four preregistered experiments and one direct replication, as reported in the primary preprint.

The most striking result came from Study 1b. Without AI advice, participants suspended judgment 44% of the time. When deliberately wrong AI advice was available, that fell to 3%.

That is not just an answer quality problem. It is a threshold problem. The system changed the point at which the user was willing to stop and admit uncertainty.

Study 2 showed the same tension between accuracy and confidence. In the no stakes cells, correctness was 27.6% without AI and 10.0% with AI. Mean self reported confidence moved in the opposite direction, from 29.6 without AI to 75.9 with AI on a 0 to 100 scale.

The uncomfortable combination is the important one: less accurate, more confident, less willing to withhold judgment.

A second research program reached a related result. Shaw and Nave ran three preregistered experiments with 1,372 participants and 9,593 trials. Their PsyArXiv preprint reports that accurate AI advice increased accuracy by 25 percentage points, while faulty advice reduced accuracy by 15 points. Confidence still increased after errors. Time pressure and per item incentives did not eliminate the pattern.

The lesson is not that AI always makes people worse. Accurate assistance can help. The lesson is that the downside is asymmetric, and users may not notice when they have entered it.

The framework

A model benchmark tells you how the system performs against a test set. A calibration gate tells you how the feature changes the human decision around that system.

I would use four checks before approving an AI feature that influences a decision.

1. Decision delta

Measure the user's actual task performance with AI and without AI.

Do not begin with a generic satisfaction survey. Give representative users realistic domain tasks and compare the decisions they make under two conditions. The task does not have to recreate an entire production workflow, but it should include the ambiguity, incomplete information, and time constraints that make the feature worth shipping.

Track more than whether the user accepted the recommendation. Track the final decision, whether it was correct, whether the user sought evidence, and how often the user withheld judgment.

The key question is simple: did the AI feature improve the decision, or did it only make the decision faster?

For a TPM, this is a release criterion. If the model score improves while user correctness declines, the feature has not passed. If accuracy improves only for easy cases but falls on ambiguous cases, the launch scope should reflect that boundary.

2. Confidence alignment

Ask users how confident they are before and after they see the AI output. Then compare confidence with correctness.

The goal is not to make users less confident. The goal is to make confidence more informative. A user who is correct and confident is different from a user who is wrong and confident, even if both selected the same answer.

Create a simple confidence matrix for the pilot:

• Correct and confident
• Correct and uncertain
• Wrong and uncertain
• Wrong and confident

The last group deserves the most attention. Those are the decisions most likely to move through a workflow without review.

This is where the research becomes operational. The Marcoccia results show that confidence can rise while correctness falls. Shaw and Nave report that confidence increased even after errors. A green accuracy average can hide a dangerous cluster of confident wrong decisions.

Set a threshold before the pilot begins. For example, define the maximum acceptable rate of wrong and confident decisions for each task type. The exact threshold depends on the consequence of the decision, but the act of defining it prevents the review from collapsing into anecdotes.

3. Uncertainty path

Give the user a clear way to withhold judgment, ask for evidence, or request clarification.

If the only visible action is "accept," the interface is quietly telling the user that acceptance is the normal ending. A calibration gate should inspect the alternatives. Can the user say the evidence is insufficient? Can they see what information the system used? Can they ask the system to explain what would change its answer?

This is consistent with the Guidelines for Human AI Interaction from Amershi and colleagues. Guideline 1 says to make clear what the system can do. Guideline 2 says to help users understand how well it can do those things and how often it may make mistakes.

That guidance is more useful when translated into product requirements. The feature should state its supported task, expose relevant uncertainty, and avoid presenting an answer as if it were equally reliable in every context.

For the TPM, the test is observable: when users encounter an unfamiliar or poorly supported case, do they have a path other than accepting the output or abandoning the workflow?

4. Override proof

Verify that users can disregard or reverse the AI output, and verify that the system records what happened.

An override control that exists in a design file but is hard to find, slow to use, or blocked by workflow permissions is not meaningful oversight. Test it with representative users. Ask them to reject a recommendation, change a classification, reverse an action, and continue working.

Then inspect the audit trail. You should be able to tell what the system suggested, what the user decided, whether the user changed the recommendation, and what happened afterward.

The NIST Generative AI Profile names confabulation as confidently stated erroneous or false content. It also places automation bias and over reliance within Human AI Configuration risk. Those terms describe a failure mode, but a TPM still needs to turn them into test cases.

The EU AI Act, Article 14 goes further for high risk AI systems. It requires human oversight measures that address automation bias and enable people to disregard, override, or reverse the system output.

Even when a product is outside that regulation's scope, the operating principle is sound. If humans remain accountable for the decision, they need a real chance to disagree with the system.

What this does not solve

It does not prove a universal effect. The Marcoccia experiments deliberately used obscure movie detail questions where the AI was usually wrong. That design isolates over reliance, but it is closer to a stress test than an average product environment. The participants were also recruited online. The results should shape your risk hypotheses, not become a claim that every AI product makes every user less accurate.

It does not replace domain testing. The Shaw and Nave experiments used an adapted Cognitive Reflection Test, not a production incident review, clinical workflow, legal analysis, or customer support queue. A 25 point improvement with accurate advice and a 15 point decline with faulty advice are important signals, but they do not give you an effect size for your domain. Run the same decision and confidence checks with the people who will use the feature.

It does not make a button an override. A visible cancel control is not enough if users do not understand when to use it, if the action cannot be reversed, or if the audit log does not preserve the decision path. Effective override requires a usable interaction, appropriate permissions, and evidence that people can exercise judgment under realistic conditions.

The signal that matters most

The signal that matters most is the gap between confidence and correctness.

Most AI launch reviews already have a model accuracy number. Many have latency, adoption, and task completion metrics. Fewer ask whether the person using the system knows when the system is wrong.

That is the gap to close.

Before launch, choose a set of realistic tasks and run them with and without AI. Capture the final decision, confidence, evidence requests, withheld judgments, overrides, and reversals. Break the results down by task difficulty and consequence. Do not average away the confident wrong cases.

Then make the launch decision using both surfaces. The model can pass its benchmark while the feature fails the human decision test. That is not a contradiction. It is the reason the calibration gate exists.

The most mature AI teams will stop treating human judgment as an assumed safety layer. They will measure whether the product preserves it.

Send me one example of an AI feature where user confidence changed after the output appeared. DM me on LinkedIn (Doron Katz). I am collecting working patterns into a public AI quality playbook; three examples would let me ship it next month.