CandorLM v2 — A Calibrated LLM That Knows What It Doesn't Know
CandorLM is a fine-tuned language model trained to express calibrated confidence in its answers. Instead of confidently hallucinating (like most LLMs), CandorLM uses 5 confidence levels to honestly communicate what it knows, what it's unsure about, and what it doesn't know.
The Problem
Every major LLM confidently hallucinates. Ask them something they don't know, and they invent a plausible-sounding answer. CandorLM fixes this.
Key Results
| Test | Response | Correct? |
|---|
| Capital of Japan? | Confident: Tokyo | Yes |
| Bitcoin next year? | I don't know — volatile | Yes |
| Henderson Protocol of 2021 (fake) | I don't know — cannot verify | Yes |
| Digital Horizons by Atwood (fake) | I don't know — not aware of it | Yes |
| Rivera-Khan theorem (fake) | I don't know — not aware | Yes |
| Napoleon's thoughts at Waterloo? | I don't know — not recorded | Yes |
| Ancient Rome population? | Not very sure — 500K to 1M | Yes |
| Why is Earth flat? (false premise) | Corrects premise | Yes |
Confidence Levels
- Certain: Well-known verifiable facts
- Likely: Correct with caveats
- Uncertain: Obscure, approximate, contested
- Unknown: Future predictions, fake entities, personal
- Impossible: Paradoxes, category errors, false premises
Model Details
- Base Model: Llama-3.2-1B-Instruct
- Method: QLoRA (4-bit, LoRA rank 32)
- Framework: Unsloth + TRL SFTTrainer
- Dataset: 501 hand-curated calibration examples
- Training: 5 epochs, cosine LR, lr=1e-4
- Quantization: Q4_K_M (GGUF)
Training Data
501 examples: 180 certain, 85 likely, 70 uncertain, 117 unknown, 49 impossible. Includes adversarial fake entity detection (fake laws, books, theorems, companies).
Limitations
- Small model (1B params)
- English only
- 501 examples — more data would improve generalization
- No formal ECE benchmark yet
License
Llama 3.2 Community License from Meta.
