This model is a fine-tuned version of mistralai/Ministral-8B-Instruct-2410 optimized for uncertainty calibration using the ConfTuner method, as presented in the paper ConfTuner: Training Large Language Models to Express Their Confidence Verbally, accepted by NeurIPS 2025.

Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as science, law, and healthcare, where accurate expressions of uncertainty are essential for reliability and trust. However, current LLMs are often observed to generate incorrect answers with high confidence, a phenomenon known as "overconfidence". Recent efforts have focused on calibrating LLMs' verbalized confidence: i.e., their expressions of confidence in text form, such as "I am 80% confident that...". Existing approaches either rely on prompt engineering or fine-tuning with heuristically generated uncertainty estimates, both of which have limited effectiveness and generalizability. Motivated by the notion of proper scoring rules for calibration in classical machine learning models, we introduce ConfTuner, a simple and efficient fine-tuning method that introduces minimal overhead and does not require ground-truth confidence scores or proxy confidence estimates. ConfTuner relies on a new loss function, tokenized Brier score, which we theoretically prove to be a proper scoring rule, intuitively meaning that it "correctly incentivizes the model to report its true probability of being correct". ConfTuner improves calibration across diverse reasoning tasks and generalizes to black-box models such as GPT-4o. Our results further show that better-calibrated confidence enables downstream gains in self-correction and model cascade, advancing the development of trustworthy LLM systems.

This model is intended for research and applications that require Large Language Models to express their confidence verbally, particularly in high-stakes domains like science, law, and healthcare where reliable uncertainty quantification is crucial. It can be used to generate text responses that include calibrated verbal confidence statements.

The model is not intended for use in production environments without rigorous domain-specific validation and bias mitigation. It should not be used in critical applications where a potential miscalibration of verbalized confidence could lead to significant adverse outcomes. Generating misleadingly confident or unconfident responses is strictly out of scope.

While ConfTuner aims to improve verbalized confidence calibration, LLMs can still exhibit biases present in their training data and may generate incorrect answers. The method does not entirely eliminate risks of overconfidence or hallucination. Thorough evaluation for specific use cases is essential.

Users are advised to validate the model's performance and calibration on their target tasks and datasets. Human oversight is recommended for critical applications to ensure responsible deployment.

To get started with the model, you can refer to the official GitHub repository for detailed instructions on dataset generation, training, and testing.

The repository provides an inference.py script for evaluating the fine-tuned model. First, clone the repository and install dependencies:

Then, you can evaluate the fine-tuned model using the inference.py script:

Replace /path/to/your/checkpoint with the path to the model checkpoint (e.g., liushiliushi/ConfTuner-Ministral) and dataset_name with your desired dataset (e.g., hotpot_qa, trivia_qa).

For testing on linguistic confidence levels (high/medium/low) or cascading performance, refer to the Testing section in the GitHub README.

The model was fine-tuned using the ConfTuner method, which introduces a novel tokenized Brier score loss function to encourage verbalized confidence calibration.

Training datasets were generated for different base models (Llama-3.1, Qwen, Mistral) and also using GPT-4o's responses. The datasets utilized include:

More details on dataset generation can be found in the GitHub repository.

ConfTuner fine-tunes LLMs using an efficient semi-supervised method. The training process leverages accelerate with deepspeed for distributed training and utilizes Parameter-Efficient Fine-Tuning (PEFT) with the LoRA method. Training commands also indicate options for consistency loss (--add_loss_con), training for coarse confidence levels (--train_coarse), and merging PEFT weights after training (--merge_peft).

Key training hyperparameters used for fine-tuning the Mistral model include:

The model's performance on verbalized confidence and cascading behavior is evaluated across various reasoning tasks.

Testing datasets include versions of:

Evaluation is performed on confidence levels ranging from 0-100, as well as qualitative linguistic confidence levels (high/medium/low). Cascading performance, often evaluated with black-box models like GPT-4o, is also a key metric.

For questions or feedback, please refer to the GitHub repository for contact information.