This is a
vicuna-7b-v1.3 model fine-tuned with
QLoRA for long-form legal question answering in
French.
We use the
Long-form Legal Question Answering (LLeQA) dataset to fine-tune the model. LLeQA is a French native dataset for studying legal information retrieval and question answering. It consists of a knowledge corpus of 27,941 statutory articles collected from the Belgian legislation, and 1,868 legal questions posed by Belgian citizens and labeled by experienced jurists with a comprehensive answer rooted in relevant articles from the corpus.
We fine-tune the model through 4-bit QLoRA finetuning with an effective batch size of 8 for 10 epochs (i.e., 1.1K steps) using paged AdamW optimizer with default momentum parameters and constant learning rate schedule of 2e-4. We employ NormalFloat4 with double quantization for the base models and add LoRA adapters on all linear layers by setting r=16, alpha=32 while utilizing float16 as computation datatype. Additionally, we perform NTK-aware scaling of RoPE to extend the context window to 4096 tokens. Training takes around 7.5 hours to complete on a single Tesla V100 GPU with 32GBs of memory. More details can be found in this
paper and
repository.
1@article{louis2023interpretable,
2 author = {Louis, Antoine and van Dijck, Gijs and Spanakis, Gerasimos},
3 title = {Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models},
4 journal = {CoRR},
5 volume = {abs/2309.17050},
6 year = {2023},
7 url = {https://arxiv.org/abs/2309.17050},
8 eprinttype = {arXiv},
9 eprint = {2309.17050},
10}
