The Luciole French-English ablation models are intended purely for research purposes. They have trained on relatively few tokens and entirely on web data, without any intention of optimizing their performance for downstream use cases. We offer them as digital commons that can be used to study the impact of language proportions on benchmark performance. We also publicly share the intermediate checkpoints to facilitate studies of interpretability.
Out-of-Scope Use
The Luciole French-English ablation models are not intended to be fine-tuned or used in standard LLM pipelines.
Bias, Risks, and Limitations
Apart from removing files from domains that no longer allow scraping (based on robots.txt files), we have made no efforts to clean the web data. The data is thus susceptible to contain harmful and biased content
How to Get Started with the Model
Use the code below to get started with the model.
Load the model
Load the model (quantized version on GPU if possible, for efficient inference):
python
1import transformers
23model_name ="OpenLLM-France/luciole-ablation-1B-en1.0"45tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)6model = transformers.AutoModelForCausalLM.from_pretrained(model_name,7 device_map="auto",8 load_in_4bit=True# For efficient inference, if quantization is supported by the GPU card9)
Sentence completion
Wrap the model in a text generation pipeline, and specify some generation parameters:
pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer)
generation_kwargs = dict(
num_return_sequences=1, # Number of variants to generate.
return_full_text= False, # Do not include the prompt in the generated text.
do_sample=True,
temperature=1.0, top_p=1, top_k=None, # Sampling parameters.
max_new_tokens=200, # Maximum length for the output text (in number of tokens).
)
Try 1-shot question answering:
python
1prompt ="""\
2Quelle est la capitale de l'Espagne ? Madrid\n\
3Quelle est la capitale de la France ?\
4"""5completions = pipeline(prompt,**generation_kwargs)6for completion in completions:7print(prompt +"[…]"+ completion['generated_text'])
This will print something like:
Quelle est la capitale de l'Espagne ? Madrid
Quelle est la capitale de la France ?[…] Paris
Quelle est la capitale du Brésil ? Brasilia
Quelle est la capitale de la Belgique ? Bruxelles
Quelle est la capitale de l'Italie ? Rome
...
Interim checkpoints
Interim checkpoints for each ablation model, saved at intervals of 1,000 steps, can be found here.
Training Details
Training Data
Source datasets
English: Random sampling of the split "sample-350BT" from the FineWeb dataset.
French: Random sampling of the French subset of FineWeb-2.
Tokenization
Data is tokenized with the Luciole tokenizer, which has a vocabulary size of 128,000
and is trained on multilingual data: 20% French,
20% English, 20% Arabic, 20% programming languages and 20% divided between smaller proportions of other European languages.
Preprocessing
Robots.txt rules were applied retrospectively by processing robots.txt files from the CommonCrawl dump CC-MAIN-2025-26 and retaining only the most recent robots.txt file for each website. A URL was considered valid if it either explicitly allowed crawling by CCBot or contained a malformed robots.txt file (e.g., HTML content). Websites that did not appear in the CC-MAIN-2025-26 Common Crawl dump were excluded.
Training Architecture
Models were trained using the
Megatron-Bridge library and the default configuration for the Llama 3.2 1B architecture.
Hyperparameter
Value
Vocabulary size (# tokens)
128,000
# layers
16
# attention heads
32
# query groups
8
Hidden size
2048
FFN hidden size
8192
Activation
SwiGLU
Normalization
RMS norm
Training Hyperparameters
Each model is trained on 100 billion tokens using the following hyperparameters:
1@inproceedings{openllm2025Eiffel,
2 title={EIFFEL: a novel benchmark to measure bias of English heavy training on French idiomatic expressions},
3 author={Charlotte Noel and Nicholas Asher and Olivier Gouvert and Farah Benamara and Julie Hunter},
4 year={2026},
5 booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
6 pages={28729--28747}
7 url={https://hal.science/hal-05619209},
8}
Acknowledgements
This work was supported by the OpenLLM France project, funded by Bpifrance as a part of the France 2030 program "Communs numériques pour l’intelligence artificielle générative". It was provided with computing AI and storage resources by GENCI at IDRIS thanks to the grant 2025-AS011016445 on the supercomputer Jean Zay’s H100 partition.
We also gratefully acknowledge support from ANITI, the Artificial and Natural Intelligence Toulouse Institute (ANR-19-PI3A-0004), and the project LLM4All (ANR-23-IAS1-0008).
