Zagreus-0.4B-fra
Zagreus-0.4B-fra is a bilingual English/French foundational Small Language Model (SLM) trained
from scratch by the
mii-llm community (
Made in Italy – Large Language Model) on the
Seeweb HPC infrastructure.
This is a base (pre-trained) model — it is not instruction-tuned and is intended for researchers, developers, and practitioners who want to fine-tune or build upon a high-quality bilingual English/French foundation.
The Zagreus family represents one of the few openly released, high-performing small language models dedicated to European Romance languages, trained entirely from first principles with a fully transparent pipeline.
Model Details
| Property | Value |
|---|
| Architecture | Modified Llama-3.2 (fully dense) |
| Parameters | ~400M |
| Hidden size | 960 |
| Intermediate size | 2560 |
| Layers | 32 |
| Attention heads | 15 (KV heads: 5) |
| Activation | SiLU |
| Context length | 4096 tokens |
| Tokenizer | Llama-3.2 (vocab_size: 128,256) |
| Positional encoding | RoPE (theta: 10000.0) |
| Tied embeddings | Yes |
| Precision | BF16 |
| Languages | English (~400B tokens), French (~400B tokens) |
| Training tokens | ~1 trillion |
| Training framework | Nanotron (mii-llm fork) |
| Infrastructure | 64× NVIDIA A100 GPUs (8 nodes × 8 GPUs), Seeweb HPC |
Training Data
All datasets used are fully open source and released by Hugging Face:
Token distribution: ~400B English + ~400B French + ~200B Code ≈ 1 trillion tokens total
Tokenization
Raw datasets were tokenized using the
Llama-3.2 tokenizer (
meta-llama/Llama-3.2-1B) via the
datatrove library. The process ran for over
three weeks of continuous computation on CPU nodes via Slurm, generating approximately 3–5 TB of tokenized data shards.
Architecture Choice
We adopted a modified Llama-3.2 fully dense architecture. The choice of a dense model over Mixture-of-Experts (MoE) in the small-parameter regime (~500M) was deliberate: in tightly constrained capacity settings, routing overhead and expert under-utilization typical of MoE architectures may offset their theoretical efficiency advantages. Dense models provide better compute utilization and more stable training dynamics at this scale.
Pre-training Configuration
Full Nanotron YAML configuration used for training:
1checkpoints:
2 checkpoint_interval: 5000
3 checkpoints_path: checkpoints_zagreus_fra
4 checkpoints_path_is_shared_file_system: false
5 save_final_state: false
6 save_initial_state: false
7data_stages:
8- data:
9 dataset:
10 dataset_folder:
11 - /training/pretraining/fineweb-fra/tokenized
12 - /training/pretraining/fineweb-edu-350BT/000_tokenized_output
13 - /training/pretraining/fineweb-edu-350BT/011_tokenized_output
14 - /training/pretraining/fineweb-edu-350BT/012_tokenized_output
15 - /training/pretraining/fineweb-edu-350BT/013_tokenized_output
16 - /training/pretraining/fineweb-edu-350BT/014_tokenized_output
17 - /training/pretraining/fineweb-edu-350BT/015_tokenized_output
18 - /training/pretraining/fineweb-edu-350BT/016_tokenized_output
19 - /training/pretraining/finepdf-fra/000_tokenized_output
20 - /training/pretraining/starcoder_tokenized/000_tokenized_output
21 num_loading_workers: 0
22 seed: 8
23 name: stable phase
24 start_training_step: 1
25general:
26 benchmark_csv_path: null
27 consumed_train_samples: null
28 ignore_sanity_checks: true
29 project: zagreus
30 run: zagreus-350M-fra
31 seed: 8
32 step: null
33logging:
34 iteration_step_info_interval: 1
35 log_level: info
36 log_level_replica: info
37model:
38 ddp_bucket_cap_mb: 100
39 dtype: bfloat16
40 init_method:
41 std: 0.03227
42 make_vocab_size_divisible_by: 1
43 model_config:
44 bos_token_id: 128000
45 eos_token_id: 128001
46 hidden_act: silu
47 hidden_size: 960
48 initializer_range: 0.02
49 intermediate_size: 2560
50 is_llama_config: true
51 max_position_embeddings: 4096
52 num_attention_heads: 15
53 num_hidden_layers: 32
54 num_key_value_heads: 5
55 pad_token_id: null
56 pretraining_tp: 1
57 rms_norm_eps: 1.0e-05
58 rope_interleaved: false
59 rope_scaling: null
60 rope_theta: 10000.0
61 tie_word_embeddings: true
62 use_cache: true
63 vocab_size: 128256
64optimizer:
65 accumulate_grad_in_fp32: true
66 clip_grad: 1.0
67 learning_rate_scheduler:
68 learning_rate: 0.003
69 lr_decay_starting_step: 750000
70 lr_decay_steps: 50000
71 lr_decay_style: linear
72 lr_warmup_steps: 4000
73 lr_warmup_style: linear
74 min_decay_lr: 1.0e-7
75 optimizer_factory:
76 adam_beta1: 0.9
77 adam_beta2: 0.95
78 adam_eps: 1.0e-08
79 name: adamW
80 torch_adam_is_fused: true
81 weight_decay: 0.01
82 zero_stage: 0
83parallelism:
84 dp: 64
85 expert_parallel_size: 1
86 pp: 1
87 pp_engine: 1f1b
88 recompute_layer: false
89 tp: 1
90 tp_linear_async_communication: true
91 tp_mode: REDUCE_SCATTER
92 tp_recompute_allgather: true
93profiler: null
94tokenizer:
95 tokenizer_max_length: null
96 tokenizer_name_or_path: meta-llama/Llama-3.2-1B
97 tokenizer_revision: null
98tokens:
99 batch_accumulation_per_replica: 1
100 limit_test_batches: 0
101 limit_val_batches: 0
102 micro_batch_size: 4
103 sequence_length: 4096
104 train_steps: 2000000
105 val_check_interval: 5000
Slurm Launch Script
1#SBATCH --job-name=350_fr
2#SBATCH --account=YOUR_ACCOUNT
3#SBATCH --partition=PARTITION
4#SBATCH --nodes=8
5#SBATCH --gres=gpu:8 # 8 A100 per node = 64 total
6#SBATCH --cpus-per-task=32
7#SBATCH --time=4-00:00:00
8#SBATCH --output=slurm-%j.out
9
10################ 0. Environment ################
11module purge
12module load profile/global
13module load python/3.11 cuda/12.2 cudnn nccl gcc
14
15source /path/to/venv/nanotron/bin/activate
16
17export HF_HOME=/path/to/hf_home
18export TRANSFORMERS_OFFLINE=1
19export HF_HUB_OFFLINE=1
20export HF_DATASETS_OFFLINE=1
21export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
22export NCCL_IB_DISABLE=0
23export NCCL_SOCKET_IFNAME="ib0,eno,eth"
24export WANDB_MODE=disabled
25
26################ 1. Distributed vars ############
27GPUS_PER_NODE=4
28NNODES=$SLURM_JOB_NUM_NODES
29NODE_RANK=$SLURM_NODEID
30MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n1)
31MASTER_PORT=29400
32RDZV_ID=$SLURM_JOB_ID
33
34################ 2. Launch ######################
35srun torchrun \
36 --nnodes $NNODES \
37 --nproc_per_node $GPUS_PER_NODE \
38 --rdzv_id $RDZV_ID \
39 --rdzv_backend c10d \
40 --rdzv_endpoint $MASTER_ADDR:$MASTER_PORT \
41 /path/to/nanotron/run_train.py \
42 --config-file smollm2/zagreus_350M_fra.yaml
Checkpoint Conversion to Hugging Face Format
1torchrun --nproc_per_node=1 -m examples.llama.convert_nanotron_to_hf \
2 --checkpoint_path=checkpoints/<step> \
3 --save_path=hf_checkpoints/<step> \
4 --tokenizer_name meta-llama/Llama-3.2-1B
Evaluation
Evaluation Commands
1lm-eval --model hf --model_args pretrained=<checkpoint> \
2 --tasks m_mmlu_fr --num_fewshot 5 --device cuda:0 --batch_size 1
3
4lm-eval --model hf --model_args pretrained=<checkpoint> \
5 --tasks hellaswag_fr,arc_fr --device cuda:0 --batch_size 1
Checkpoint Progression
The table below tracks benchmark scores across training checkpoints, demonstrating steady model improvement throughout pre-training:
| Checkpoint | MMLU FR ↑ | ARC FR ↑ | HellaSwag FR ↑ | Average |
|---|
| 129k | 0.262 | — | — | 0.262 |
| 231k | 0.263 | — | — | 0.263 |
| 365k | 0.256 | 0.278 | 0.414 | 0.316 |
| 456k | 0.267 | — | — | 0.267 |
| 603k | 0.256 | 0.278 | 0.414 | 0.316 |
| 705k | 0.266 | 0.281 | 0.417 | 0.321 |
Best overall checkpoint: 705k with an average of 0.321 across all three benchmarks.
Usage
This is a base model — it performs causal language modelling (text completion) and is not instruction-tuned. It is best suited as a starting point for fine-tuning.
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
3
4model_id = "mii-llm/zagreus-0.4B-fra"
5
6tokenizer = AutoTokenizer.from_pretrained(model_id)
7model = AutoModelForCausalLM.from_pretrained(
8 model_id,
9 torch_dtype=torch.bfloat16,
10 device_map="auto"
11)
12
13# Base model: text completion, not instruction following
14prompt = "L'intelligence artificielle est une discipline qui"
15
16inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
17
18output = model.generate(
19 **inputs,
20 max_new_tokens=200,
21 temperature=0.8,
22 do_sample=True,
23 repetition_penalty=1.1
24)
25
26print(tokenizer.decode(output[0], skip_special_tokens=True))
Full Model Family
Base Models (Zagreus)
Post-trained Models (Nesso) — English/Italian
Citation
If you use this model in your research, please cite:
1@misc{zagreus2025,
2 title = {The Joy and Pain of Training an LLM from Scratch:
3 A Technical Report on the Zagreus and Nesso Model Families},
4 author = {mii-llm community},
5 year = {2025},
6 howpublished = {\url{https://github.com/mii-llm/zagreus-nesso-slm}},
7}
Acknowledgements
- Antonio Baldassarra (CEO, Seeweb) and Marco Cristofanilli (Head of AI, Seeweb) for commissioning and sponsoring the infrastructure
- The Hugging Face team for Nanotron, datatrove, FineWeb, FineWeb-2, and FinePDFs
- The mii-llm open-source community for contributions to multilingual evaluation harnesses and the Nanotron fork
License
Released under the Apache 2.0 license.
Made with ❤️ in Italy by
mii-llm
