Gen AI

LLM Confidence & Calibration

A well-calibrated model knows what it doesn't know. If it says it is 80% sure, it should be right exactly 80% of the time. Unfortunately, modern training techniques often destroy this self-awareness, making models arrogant and wrong.

LLM calibration compares the model's stated confidence in its answer against the actual probability that the answer is correct

Why Does This Exist?

When building reliable software on top of AI, it is highly useful to know how confident the model is. If an AI summarizes a medical document and attaches a 99% confidence score, you might automate the workflow. If it attaches a 40% confidence score, you route it to a human doctor for review.

Calibration is the statistical measure of how honest a model's confidence scores are. If a perfectly calibrated model says it is 80% confident on 100 different questions, it will get exactly 80 of those questions right.

The crisis in modern GenAI is that foundational base models are usually well-calibrated, but the safety and alignment training (RLHF) required to make them usable as chatbots utterly destroys their calibration. Modern LLMs are notoriously arrogant—they will declare they are 100% confident in a hallucination that is entirely wrong.

Think of It Like This

The honest student vs the politician

A Base Model is an honest student taking a multiple-choice test. If they don't know the answer, they leave the confidence bubble blank. If you grade their test, their confidence perfectly matches their accuracy.

An RLHF Fine-Tuned Model is a politician on a debate stage. They have been heavily rewarded by focus groups for sounding authoritative, decisive, and helpful at all times. If they don't know a fact, looking unsure makes them look weak, so they confidently make something up. They are rewarded for the presentation of confidence, completely detaching their tone from their actual accuracy.

How It Actually Works

There are two primary ways to measure an LLM's confidence:

1. Verbalized Confidence (Prompting)

You simply ask the model: "Answer this question, and tell me how confident you are from 0% to 100%." Because of RLHF, this method is almost entirely useless. The model is highly incentivized to output strings like "100%" or "I am absolutely certain" because human raters during the training phase preferred AI assistants that sounded confident and helpful. The model's verbalized confidence is essentially a hallucinated number.

2. Logprobs (Mathematical Confidence)

Instead of asking the model, you look directly at its internal math. When an LLM generates a token, it outputs a probability distribution. If the top token has a probability of 0.95 (95%), the model is mathematically 95% confident in that specific word.

APIs often expose these as logprobs. While much more accurate than verbalized confidence, RLHF still degrades them. RLHF pushes the model's mathematical probabilities toward the extremes (0% or 100%) to force decisive generation, stripping away the nuanced middle ground (e.g., 40%) that represents true uncertainty.

Semantic Entropy

To get around broken calibration, engineers use advanced techniques like Semantic Entropy. Instead of asking for a confidence score, you ask the model the same question 10 times at a high temperature.

If the model gives you the exact same semantic answer 10 times (even phrased differently), it is truly confident.
If it gives you 4 completely different answers, it is mathematically guessing, regardless of how authoritative its tone is.

Show Me the Code

This demonstrates how checking the raw logprobs from an API is a much safer way to gauge true confidence than reading the model's text.

1# Conceptual example of reading API logprobs for confidence2response = client.chat.completions.create(3    model="gpt-4",4    messages=[{"role": "user", "content": "What is the capital of Australia?"}],5    logprobs=True,      # Request the raw math!6    top_logprobs=27)8
9# The model's text response10answer = response.choices[0].message.content # "Canberra"11
12# The mathematical confidence for the word "Canberra"13token_math = response.choices[0].logprobs.content[0]14confidence = math.exp(token_math.logprob) # Convert logprob to percentage15
16print(f"Answer: {answer}")17print(f"Mathematical Confidence: {confidence * 100:.2f}%")18
19# Even if the text said "I THINK it might be Canberra", 20# the math might show 99.9% confidence. Trust the math.

Watch Out For

Tone is the enemy of truth

Never, under any circumstances, use an LLM's tone as a proxy for its accuracy. The model is not a human; it does not stutter or use filler words when it is unsure. It will generate a completely fabricated legal citation with the exact same crisp, professional, authoritative tone as a factual one.

The Quick Version

Calibration measures whether a model's stated confidence matches its actual accuracy.
Base models (before safety training) are generally well-calibrated.
RLHF (human preference training) destroys calibration, teaching the model to sound confident and authoritative even when it is completely wrong.
Verbalized confidence (asking the model how sure it is) is essentially useless due to this training.
Logprobs (looking at the raw token probabilities) provide a much more honest measure of the model's internal uncertainty.
The safest way to measure true confidence is to generate multiple answers and see if the model contradicts itself.