This is the intermediate ContextCite supervised fine-tuning (CC-SFT) checkpoint used in the fully self-supervised setting of the SelfCite paper. It is initialized from Llama-3.1-8B-Instruct and trained on SFT data generated with ContextCite using 256 ContextCite calls.

This checkpoint is the model before SimPO preference optimization. For the resulting SelfCite model after SimPO, use voidism/SelfCite-8B-from-CC.

Please refer to the SelfCite repository for training and usage details.

Training details

Base model: meta-llama/Llama-3.1-8B-Instruct
Training examples: 11,185
Learning rate: 7e-6
Epochs: 1
Per-device batch size: 1
Number of devices: 8
Effective batch size: 8
Scheduler: cosine with a 0.1 warmup ratio
Precision: bfloat16
Transformers: 4.44.2
PyTorch: 2.4.1+cu121

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "voidism/SelfCite-8B-CC-SFT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

Citation

Please cite SelfCite and LongCite if they are helpful to your work.

bibtex

@inproceedings{chuang2025selfcite,
  title={SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models},
  author={Yung-Sung Chuang and Benjamin Cohen-Wang and Shannon Zejiang Shen and Zhaofeng Wu and Hu Xu and Xi Victoria Lin and James Glass and Shang-Wen Li and Wen-tau Yih},
  journal={arXiv preprint arXiv:2502.09604},
  year={2025}
}

@article{zhang2024longcite,
  title={LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA},
  author={Jiajie Zhang and Yushi Bai and Xin Lv and Wanjun Gu and Danqing Liu and Minhao Zou and Shulin Cao and Lei Hou and Yuxiao Dong and Ling Feng and Juanzi Li},
  journal={arXiv preprint arXiv:2409.02897},
  year={2024}
}