Luciole-8B-Base is a pretrained 8B parameter causal language model with a hybrid Mamba-transformer architecture. It was developed by LINAGORA and the OpenLLM-France consortium as a part of the OpenLLM France project, funded by BPI France through the France 2030 program.
It was trained on the GENCI supercomputer Jean Zay, managed by IDRIS.
Luciole-8B-Base was trained on around 5 trillion tokens of multilingual data, including English (41.9%), French (30.4%), German (3.8%), Spanish (3.5%), Italian (1.9%), Portuguese (1.3%), Dutch (1.0%), Arabic (0.5%), and a small subset of regional languages including regional languages of the French metropolitan area, French variants, and French creoles from around the world (0.4%).
The latter were selected from the FineWeb 2 dataset and include Basque, Breton, Catalan, Corsican, Franco-Provençal, Guadeloupean Creole French, Guianese Creole French, Occitan, Picard, Réunion Creole French, Saint Lucian Creole French, Seselwa Creole French, Tahitian, and Walloon.
Training data also include parallel data from a selection of languages (1.7%), as well as several programming languages (9.2%) and English mathematical data (3.5%).
Luciole-8B-Base is a foundation language model trained solely to predict the most probable next word in a sequence. It is designed as the first brick in a more complex training pipeline that would include multitask training on diverse instructions or focused fine-tuning on select downstream tasks, as well as possible alignment for human preferences.
Downstream use
Due to its multilingual training, Luciole-8B-Base can be fine-tuned for downstream tasks centered on the generation of multilingual text, with a special focus on French and English.
Out-of-Scope Use
Luciole-8B-Base is not intended to generate text directly for end use cases. It must be fine-tuned first. Its pretraining is optimized for multilingual performance, especially in French and English, and might perform less well on other languages without additional training. While trained on code data, it is not optimized for code generation tasks.
Bias, Risks, and Limitations
Like other foundation models, Luciole-8B-Base is trained on large amounts of web data. Additionally, due to the scarcity of French textual non-web data published under open licenses, much of our French data comes from older works in the public domain that carry biases from other time periods. While we made efforts to reduce toxic and offensive content in the Luciole Training Dataset, Luciole-8B-Base may still generate such content. Filtering of the Luciole Training Dataset is an ongoing project to which we welcome contributions.
Recommendations
To limit the generation of undesirable content, it is advised to fine-tune Luciole-8B-Base through instruction and preference tuning (DPO, RLHF, etc.).
Example Code in Python
Load the model
Load the model (quantized version on GPU if possible, for efficient inference):
python
1import transformers
23model_name ="OpenLLM-France/Luciole-8B-Base"45tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)6model = transformers.AutoModelForCausalLM.from_pretrained(model_name,7 device_map="auto",8 load_in_4bit=True# For efficient inference, if quantization is supported by the GPU card9)
Sentence completion
Wrap the model in a text generation pipeline, and specify some generation parameters:
pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer)
generation_kwargs = dict(
num_return_sequences=1, # Number of variants to generate.
return_full_text= False, # Do not include the prompt in the generated text.
do_sample=True,
temperature=1.0, top_p=1, top_k=None, # Sampling parameters.
max_new_tokens=200, # Maximum length for the output text (in number of tokens).
)
Try 1-shot question answering:
python
1prompt ="""\
2Quelle est la capitale de l'Espagne ? Madrid\n\
3Quelle est la capitale de la France ?\
4"""5completions = pipeline(prompt,**generation_kwargs)6for completion in completions:7print(prompt +"[…]"+ completion['generated_text'])
This will print something like:
Quelle est la capitale de l'Espagne ? Madrid
Quelle est la capitale de la France ?[…] Paris
Quelle est la capitale du Brésil ? Brasilia
Quelle est la capitale de la Belgique ? Bruxelles
Quelle est la capitale de l'Italie ? Rome
...
Loading Intermediate Checkpoints
Intermediate checkpoints are released under dedicated revision tags at regular intervals throughout training:
Every 1,000 steps during the first 5,000 steps
Then every 5,000 steps up to 30,000 steps
Then every 10,000 steps beyond that
In addition, a checkpoint is provided at the end of each training phase
The total cumulative number of training steps and training tokens for each checkpoint is specified in
the YAML header of each README.md file, and in
the config.json file (under the keys "training_steps" and "training_tokens").
Training Details
Training Data
The training dataset used for the pretraining of Luciole-8B-Base is available
at OpenLLM-France/Luciole-Training-Dataset. Information on data preprocessing can be found on the data card or in the Luciole-Training repository.
Pretraining consisted of three principal phases of training with a context length of 4,096 tokens. The token breakdowns for the three phases are as follows:
Initial pretraining: 3.5 trillion tokens of diverse data
Continued pretraining: 1 trillion tokens introducing higher quality data and increasing math and code proportions
Annealing phase: 0.5 trillion tokens introducing some instruction-style and reasoning data
Pretraining was followed by one short mid-training phases to extend the context length to 131,072 tokens:
Context extension: 100 billion tokens to extend context length from 4,096 to 131,072 tokens
Training Procedure
Luciole-8B-Base is a causal decoder-only model trained on a causal language modeling task (i.e., predict the next token).
It was pre-trained on 128 - 256 H100 80GB GPUs (32 - 64 nodes) for about 237,036 GPU hours (870 hours) on the Jean Zay supercomputer.
The architecture of Luciole-8B-Base is a custom adaptation of the NemotronH-8B recipe.
It has exactly 8.08 billion free parameters,
with the following hyperparameters:
Hyperparameter
Value
Vocabulary size (# tokens)
128,000
# blocks
52
# attention heads
32
# key-value heads
8
Hidden size
4096
Intermediate size
21504
SSM State size
128
Mamba # heads
128
MLP Activation
relu2
Mamba Activation
silu
Training Hyperparameters
The details of the intitial pretraining phase are listed below. For each subsequent phase, only the values that differ from the intitial pretraining phase are listed.
The following figure shows the training loss curve for the different training phases:
metadata/training_logs/ConvergenceCurve.png
Evaluation
During the training of Luciole-8B-Base, we conducted multiple evaluations to assess performance on standard benchmarks.
The primary evaluation languages were French and English, with additional evaluations in German, Spanish, Italian, Portuguese, Dutch, and Arabic.
These evaluations were performed on intermediate checkpoints throughout training and on the final checkpoint.
We also evaluated the final checkpoint using the RULER benchmark to measure long-context performance.
For comparison, we evaluated the model against the following models, which have between 8B and 2B parameters and were trained on multilingual data, including French:
The main results are summarized in the figures below.
Click on a figure to view the complete evaluation results with detailed metrics.
The figures illustrate the evolution of evaluation performance across training checkpoints.
The x-axis corresponds to the cumulative number of training tokens, while the y-axis reports the average performance across tasks within each category.
The RULER figure differs in that it reports average performance across different context lengths.
We gratefully acknowledge BPI France for funding the OpenLLM France project under the call "Communs numériques pour l’intelligence artificielle générative" ("Digital commons for generative artificial intelligence") as a part of the France 2030 program.
Training of Luciole-8B-Base was made possible by computing AI and storage resources by GENCI at IDRIS thanks to the grant 2025-AS011016445 on the supercomputer Jean Zay’s H100 partition. We gratefully acknowledge support from GENCI and IDRIS and from Stephane Requena (GENCI) and Pierre-François Lavallée (IDRIS) in particular.
Luciole-8B-Base was created by members of LINAGORA for the OpenLLM-France project, including in alphabetical order:
Audran Bert
Akshay Chaturvedi
Olivier Gouvert
Julie Hunter
Jean-Pierre Lorré
Jérôme Louradour
Charlotte Noel
Kate Thompson
We thank the support team from IDRIS for technical guidance throughout the project, especially:
Martin Comminges (IDRIS)
Rémi Lacroix (IDRIS)
Myriam Peyrounette (IDRIS)
as well as the support team from NVIDIA, especially:
We would also like to thank members of the Gaperon, Salamandra and Apertus projects for sharing their insights with us. We also acknowledge the numerous open source actors whose resources have guided us throughout the training process, with particular thanks to Nvidia, Hugging Face and Allen AI.
Finally, we thank the entire OpenLLM-France community, whose members have helped in diverse ways.
