Lucie-7B-Instruct-human-data is a fine-tuned version of Lucie-7B, an open-source, multilingual causal language model created by OpenLLM-France.
Lucie-7B-Instruct-human-data is fine-tuned on human-produced instructions collected either from open annotation campaigns or by applying templates to extant datasets. The performance of Lucie-7B-Instruct-human-data falls below that of Lucie-7B-Instruct-v1.1; the interest of the model is to show what can be done to fine-tune LLMs to follow instructions without appealing to third party LLMs.
Note that Lucie-7B-Instruct-human-data is optimized for the generation of French text. It has not been trained for code generation or optimized for math. Such capacities can be improved through further fine-tuning and alignment with methods such as DPO, RLHF, etc.
While Lucie-7B-Instruct-human-data is trained on sequences of 4096 tokens, its base model, Lucie-7B has a context size of 32K tokens. Based on Needle-in-a-haystack evaluations, Lucie-7B-Instruct-human-data maintains the capacity of the base model to handle 32K-size context windows.
Training details
Training data
Lucie-7B-Instruct-human-data is trained on the following datasets published by third parties:
Filtering by language: Aya Dataset, Dolly and Open Assistant were filtered to keep only languages on which Lucie-7B was trained.
Filtering by keyword: Examples containing assistant responses were filtered out from Open Assistant if the responses contained a keyword from the list filter_strings. This filter is designed to remove examples in which the assistant is presented as model other than Lucie (e.g., ChatGPT, Gemma, Llama, ...).
Instruction template:
Lucie-7B-Instruct-human-data was trained on the chat template from Llama 3.1 with the sole difference that <|begin_of_text|> is replaced with <s>. The resulting template:
<s><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
Give me three tips for staying in shape.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
1. Eat a balanced diet and be sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule.<|eot_id|>
Training procedure
The model architecture and hyperparameters are the same as for Lucie-7B during the annealing phase with the following exceptions:
context length: 4096*
batch size: 1024
max learning rate: 3e-5
min learning rate: 3e-6
*As noted above, while Lucie-7B-Instruct is trained on sequences of 4096 tokens, it maintains the capacity of the base model, Lucie-7B, to handle context sizes of up to 32K tokens.
1@misc{openllm2025lucie,
2 title={The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation},
3 author={Olivier Gouvert and Julie Hunter and Jérôme Louradour and Christophe Cerisara and Evan Dufraisse and Yaya Sy and Laura Rivière and Jean-Pierre Lorré and OpenLLM-France community},
4 year={2025},
5 eprint={2503.12294},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2503.12294},
9}
Acknowledgements
This work was performed using HPC resources from GENCI–IDRIS (Grant 2024-GC011015444). We gratefully acknowledge support from GENCI and IDRIS and from Pierre-François Lavallée (IDRIS) and Stephane Requena (GENCI) in particular.
Lucie-7B was created by members of LINAGORA and the OpenLLM-France community, including in alphabetical order:
Olivier Gouvert (LINAGORA),
Ismaïl Harrando (LINAGORA/SciencesPo),
Julie Hunter (LINAGORA),
Jean-Pierre Lorré (LINAGORA),
Jérôme Louradour (LINAGORA),
Michel-Marie Maudet (LINAGORA), and
Laura Rivière (LINAGORA).
We thank
Clément Bénesse (Opsci),
Christophe Cerisara (LORIA),
Émile Hazard (Opsci),
Evan Dufraisse (CEA),
Guokan Shang (MBZUAI),
Joël Gombin (Opsci),
Jordan Ricker (Opsci),
and
Olivier Ferret (CEA)
for their helpful input.
Finally, we thank the entire OpenLLM-France community, whose members have helped in diverse ways.
