Language models sometimes generate confident-sounding text that is completely wrong. This phenomenon, called hallucination, is one of the biggest barriers to deploying AI in high-stakes applications. Users have no easy way to know when to trust a model's output and when to verify it. We need a reliability signal that flags uncertain outputs before they cause harm.
2. Challenge
Detecting hallucinations is difficult because the model itself doesn't know what it doesn't know. Traditional approaches require large labeled datasets of correct and incorrect outputs, which are expensive to create and domain-specific. The internal representations of language models are high-dimensional and difficult to interpret, making it unclear which signals correlate with reliability. Furthermore, uncertainty manifests differently across question types—a model might be calibrated for factual recall but overconfident on reasoning tasks.
3. Proposed Solution
We developed an uncertainty estimation head that measures internal consistency signals within the model during inference. The core insight is that confident predictions produce aligned internal representations, while uncertain predictions create noisy, contradictory signals. We train a small predictor network to learn the expected transformation between intermediate hidden states—deviations from this learned pattern indicate out-of-distribution inputs. This approach requires no hallucination labels; the model learns uncertainty purely from its own behavior on diverse prompts. The result is a single "I Don't Know" (IDK) score from 0-100 that can be computed alongside any generation.
4. Method
Intuition
Think of the IDK score as measuring how "surprised" the model is by its own internal processing. When answering a question it knows well, the model's internal signals flow smoothly and predictably from layer to layer. When guessing, these signals become erratic—different parts of the model disagree, the flow between layers deviates from normal patterns, and the output distribution spreads across many possible tokens. We measure all three of these signals and combine them into a single uncertainty score.
Technical Details
The IDK score combines three complementary signals:
Flow Consistency (70% weight)
A trained MLP predicts z₁₂ from z₈. High prediction error indicates unusual internal dynamics—the model is operating outside its comfort zone.
Output Entropy (20% weight)
Shannon entropy of output probabilities. High entropy means probability mass spreads across many tokens rather than one confident choice.
Head Disagreement (10% weight)
Variance across the 8 attention heads. When confident, heads converge; when uncertain, they attend to different aspects.
Training
The IDK head (~2.4M parameters) trains unsupervised on the model's own generations. We extracted ~385,000 (z₈, z₁₂) pairs from 20,000 diverse prompts across multiple datasets:
Source
Samples
Category
NaturalQuestions
77,229
Factual
HotpotQA
76,512
Multi-hop reasoning
TriviaQA
75,493
Factual trivia
SQuAD 2.0
62,840
Factual + unanswerable
Subjective
16,060
Opinions
Future/Impossible
31,180
High uncertainty
TruthfulQA
5,867
Hallucination-prone
The loss combines flow MSE, category calibration, and diversity regularization. No hallucination labels required.
LFM2 is a hybrid architecture with 10 convolution blocks and 6 grouped query attention blocks:
Layer: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Type: C C A C C A C C A C A C A C A C
▲ ▲
z₈ z₁₂
(extract) (extract)
C = Convolution (Lfm2ShortConv)
A = Attention (Lfm2Attention, GQA with 8 KV heads)
1# Prepare input2prompt ="<|startoftext|><|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"3input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)45# Get output with IDK score6outputs = model(input_ids, output_idk_score=True)78print(f"IDK Score: {outputs.idk_score.item():.1f}/100")9print(f"Components: {outputs.idk_components}")
Output Format
python
1outputs.logits # Standard LM logits [batch, seq, vocab]2outputs.idk_score # Uncertainty score [batch] (0-100)3outputs.idk_components # Dict: flow_error, head_disagreement, entropy_signal4outputs.past_key_values # KV cache for continued generation
8. Attribution
Model Development: Age van de Mei Base Model:LiquidAI for LFM2-1.2B Infrastructure:HuggingFace transformers Training: Google Colab A100
Citation
bibtex
1@misc{lfm2idk2025,
2 title={LFM2-IDK: Uncertainty Estimation via Internal Consistency},
3 author={van de Mei, Age},
4 year={2025},
5 url={https://huggingface.co/agevdmei/LiquidAI-LMF2-1.2B-plus-model-BS-detector}
6}
