Why AI overconfidence harms reliability—and how to fix it

Uncertainty is a human trait. For artificial intelligence, it’s often absent altogether.

Today’s most advanced reasoning models share a troubling habit: they deliver every answer with unwavering confidence, regardless of whether the response is correct or a random guess. This isn’t just a quirk—it’s a fundamental flaw baked into how these systems are trained.

Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have identified the root cause of this overconfidence and developed a method to correct it without sacrificing accuracy. Their approach, called Reinforcement Learning with Calibration Rewards (RLCR), trains models not only to provide answers but also to quantify their own uncertainty by outputting a confidence score alongside each response.

In extensive tests across multiple benchmarks, RLCR slashed calibration errors by up to 90% while maintaining or even improving accuracy—both on familiar tasks and entirely new challenges the model had never encountered. The findings will be presented at the upcoming International Conference on Learning Representations.

The hidden flaw behind AI’s overconfidence

The issue stems from a critical gap in traditional reinforcement learning (RL) methods, which power many of today’s top AI reasoning systems, including models like OpenAI’s o1. These systems are rewarded solely for correctness and penalized only for errors. There’s no incentive to acknowledge uncertainty or express doubt.

This creates a perverse dynamic: a model that stumbles upon the right answer through careful deduction receives the same reward as one that guesses correctly by chance. Over time, this trains models to confidently answer every question, even when their certainty is baseless. The result? A system that claims to be 95% sure of an answer when it’s actually only half right—far more dangerous than one that simply admits ignorance.

"The standard training approach is simple and effective, but it gives models no reason to say, I don’t know," explains Mehul Damani, an MIT PhD student and co-lead author of the research. "So they default to guessing, even when they’re unsure."

A mathematical fix for calibration

RLCR addresses this by introducing a single, critical adjustment to the reward function: a Brier score, a statistical measure that penalizes the difference between a model’s stated confidence and its actual accuracy. During training, models learn to reason about both the problem and their own uncertainty, producing an answer and a confidence estimate in tandem.

This dual-output approach ensures that confidently wrong answers are penalized, while unnecessarily uncertain correct ones are also discouraged. The team mathematically proved that this reward structure guarantees models that are both accurate and well-calibrated.

To validate their method, the researchers tested RLCR on a 7-billion-parameter model across a range of question-answering and mathematical benchmarks, including six datasets the model had never seen during training. The results were clear:

• Standard RL training worsened calibration compared to the base model, making models less reliable at assessing their own uncertainty.
• RLCR reversed this effect, drastically improving calibration without sacrificing accuracy.
• The approach outperformed post-hoc methods, where a separate classifier is trained to assign confidence scores after the model has already answered.

"What’s most surprising is that ordinary RL training doesn’t just fail to improve calibration—it actively damages it," says Isha Puri, another MIT PhD student and co-lead author. "The models become more capable, but also more overconfident."

Confidence estimates that actually work

The benefits of RLCR extend beyond training. At inference time, the confidence estimates generated by the model prove practically useful.

For example:

• When models produce multiple candidate answers, selecting the one with the highest self-reported confidence—or weighting responses by their confidence in a majority-voting scheme—improves both accuracy and calibration as computational resources scale.
• The act of reasoning about uncertainty itself carries value. The researchers trained classifiers on the model’s outputs and found that including the model’s explicit uncertainty reasoning as input improved the classifier’s performance, particularly for smaller models. The model’s self-reflective process contains real, actionable information—not just decorative noise.

Building trustworthy AI systems

The implications of this research stretch beyond technical benchmarks. In fields like medicine, law, and finance—where decisions carry high stakes—AI systems must do more than provide answers. They must also convey the limits of their knowledge.

RLCR represents a step toward systems that don’t just perform well, but also communicate their certainty realistically. By bridging the gap between confidence and accuracy, it could help prevent the kind of silent failures that arise when users trust AI outputs blindly.

Looking ahead, the team plans to explore how this approach scales with even larger models and more complex tasks. The goal isn’t just smarter AI—it’s AI we can trust.