Views
No views yet
python3 run_regression_experiment.py --model-identifier gpt-3.5-turbo-0125 --dataset-name trivia_qa --device cuda:0 --num-training-steps 600 --num-in-context-samples 10 --data-dir $data_dir --model-save-dir $model_save_dir --use-binary-targets --result-dir $result_dir --lr 0.00001222 --weight-decay 0.0009894 --push-to-hub1@inproceedings{ulmer-etal-2024-calibrating,
2 title = "Calibrating Large Language Models Using Their Generations Only",
3 author = "Ulmer, Dennis and
4 Gubri, Martin and
5 Lee, Hwaran and
6 Yun, Sangdoo and
7 Oh, Seong",
8 editor = "Ku, Lun-Wei and
9 Martins, Andre and
10 Srikumar, Vivek",
11 booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
12 month = aug,
13 year = "2024",
14 address = "Bangkok, Thailand",
15 publisher = "Association for Computational Linguistics",
16 url = "https://aclanthology.org/2024.acl-long.824",
17 doi = "10.18653/v1/2024.acl-long.824",
18 pages = "15440--15459",
19 abstract = "As large language models (LLMs) are increasingly deployed in user-facing applications, building trust and maintaining safety by accurately quantifying a model{'}s confidence in its prediction becomes even more important. However, finding effective ways to calibrate LLMs{---}especially when the only interface to the models is their generated text{---}remains a challenge. We propose APRICOT (Auxiliary prediction of confidence targets): A method to set confidence targets and train an additional model that predicts an LLM{'}s confidence based on its textual input and output alone. This approach has several advantages: It is conceptually simple, does not require access to the target model beyond its output, does not interfere with the language generation, and has a multitude of potential usages, for instance by verbalizing the predicted confidence or using it to re-prompting the LLM to accurately reflecting its uncertainty. We show how our approach performs competitively in terms of calibration error for white-box and black-box LLMs on closed-book question-answering to detect incorrect LLM answers.",
20}