Zagreus-0.4B-ita
Zagreus-0.4B-ita is a bilingual English/Italian foundational Small Language Model (SLM) trained
from scratch by the
mii-llm community (
Made in Italy – Large Language Model) on the
Seeweb HPC infrastructure.
This is a
base (pre-trained) model — it is not instruction-tuned and is intended for researchers, developers, and practitioners who want to fine-tune or build upon a high-quality bilingual English/Italian foundation. It serves as the base for the entire
Nesso model family.
The Zagreus family represents one of the few openly released, high-performing small language models dedicated to European Romance languages, trained entirely from first principles with a fully transparent pipeline.
Model Details
| Property | Value |
|---|
| Architecture | Modified Llama-3.2 (fully dense) |
| Parameters | ~400M |
| Hidden size | 960 |
| Intermediate size | 2560 |
| Layers | 32 |
| Attention heads | 15 (KV heads: 5) |
| Activation | SiLU |
| Context length | 4096 tokens |
| Tokenizer | Llama-3.2 (vocab_size: 128,256) |
| Positional encoding | RoPE (theta: 10000.0) |
| Tied embeddings | Yes |
| Precision | BF16 |
| Languages | English (~400B tokens), Italian (~400B tokens) |
| Training tokens | ~1 trillion |
| Training framework | Nanotron (mii-llm fork) |
| Infrastructure | 64× NVIDIA A100 GPUs (8 nodes × 8 GPUs), Seeweb HPC |
Training Data
All datasets used are fully open source and released by Hugging Face:
Token distribution: ~400B English + ~400B Italian + ~200B Code ≈ 1 trillion tokens total
Tokenization
Raw datasets were tokenized using the
Llama-3.2 tokenizer (
meta-llama/Llama-3.2-1B) via the
datatrove library. The process ran for over
three weeks of continuous computation on CPU nodes via Slurm, generating approximately 3–5 TB of tokenized data shards.
Architecture Choice
We adopted a modified Llama-3.2 fully dense architecture. The choice of a dense model over Mixture-of-Experts (MoE) in the small-parameter regime (~500M) was deliberate: in tightly constrained capacity settings, routing overhead and expert under-utilization typical of MoE architectures may offset their theoretical efficiency advantages. Dense models provide better compute utilization and more stable training dynamics at this scale.
Pre-training Configuration
Full Nanotron YAML configuration used for training:
1checkpoints:
2 checkpoint_interval: 5000
3 checkpoints_path: checkpoints_zagreus_ita_v2
4 checkpoints_path_is_shared_file_system: false
5 resume_checkpoint_path: /training/pretraining/nanotron/checkpoints_zagreus_ita_v2/630000
6 save_final_state: false
7 save_initial_state: false
8data_stages:
9- data:
10 dataset:
11 dataset_folder:
12 - /training/pretraining/fineweb-ita/tokenized
13 - /training/pretraining/fineweb-edu-350BT/000_tokenized_output
14 - /training/pretraining/fineweb-edu-350BT/011_tokenized_output
15 - /training/pretraining/fineweb-edu-350BT/012_tokenized_output
16 - /training/pretraining/fineweb-edu-350BT/013_tokenized_output
17 - /training/pretraining/fineweb-edu-350BT/014_tokenized_output
18 - /training/pretraining/fineweb-edu-350BT/015_tokenized_output
19 - /training/pretraining/fineweb-edu-350BT/016_tokenized_output
20 - /training/pretraining/finepdf-ita/000_tokenized_output
21 - /training/pretraining/starcoder_tokenized/000_tokenized_output
22 num_loading_workers: 0
23 seed: 8
24 name: stable phase
25 start_training_step: 1
26general:
27 benchmark_csv_path: null
28 consumed_train_samples: null
29 ignore_sanity_checks: true
30 project: zagreus
31 run: zagreus-350M
32 seed: 8
33 step: null
34logging:
35 iteration_step_info_interval: 1
36 log_level: info
37 log_level_replica: info
38model:
39 ddp_bucket_cap_mb: 100
40 dtype: bfloat16
41 init_method:
42 std: 0.03227
43 make_vocab_size_divisible_by: 1
44 model_config:
45 bos_token_id: 128000
46 eos_token_id: 128001
47 hidden_act: silu
48 hidden_size: 960
49 initializer_range: 0.02
50 intermediate_size: 2560
51 is_llama_config: true
52 max_position_embeddings: 4096
53 num_attention_heads: 15
54 num_hidden_layers: 32
55 num_key_value_heads: 5
56 pad_token_id: null
57 pretraining_tp: 1
58 rms_norm_eps: 1.0e-05
59 rope_interleaved: false
60 rope_scaling: null
61 rope_theta: 10000.0
62 tie_word_embeddings: true
63 use_cache: true
64 vocab_size: 128256
65optimizer:
66 accumulate_grad_in_fp32: true
67 clip_grad: 1.0
68 learning_rate_scheduler:
69 learning_rate: 0.003
70 lr_decay_starting_step: 750000
71 lr_decay_steps: 50000
72 lr_decay_style: linear
73 lr_warmup_steps: 4000
74 lr_warmup_style: linear
75 min_decay_lr: 1.0e-7
76 optimizer_factory:
77 adam_beta1: 0.9
78 adam_beta2: 0.95
79 adam_eps: 1.0e-08
80 name: adamW
81 torch_adam_is_fused: true
82 weight_decay: 0.01
83 zero_stage: 0
84parallelism:
85 dp: 64
86 expert_parallel_size: 1
87 pp: 1
88 pp_engine: 1f1b
89 recompute_layer: false
90 tp: 1
91 tp_linear_async_communication: true
92 tp_mode: REDUCE_SCATTER
93 tp_recompute_allgather: true
94profiler: null
95tokenizer:
96 tokenizer_max_length: null
97 tokenizer_name_or_path: meta-llama/Llama-3.2-1B
98 tokenizer_revision: null
99tokens:
100 batch_accumulation_per_replica: 1
101 limit_test_batches: 0
102 limit_val_batches: 0
103 micro_batch_size: 4
104 sequence_length: 4096
105 train_steps: 2000000
106 val_check_interval: 5000
Slurm Launch Script
1#SBATCH --job-name=350_it
2#SBATCH --account=YOUR_ACCOUNT
3#SBATCH --partition=PARTITION
4#SBATCH --nodes=8
5#SBATCH --gres=gpu:8 # 8 A100 per node = 64 total
6#SBATCH --cpus-per-task=32
7#SBATCH --time=4-00:00:00
8#SBATCH --output=slurm-%j.out
9
10################ 0. Environment ################
11module purge
12module load profile/global
13module load python/3.11 cuda/12.2 cudnn nccl gcc
14
15source /path/to/venv/nanotron/bin/activate
16
17export HF_HOME=/path/to/hf_home
18export TRANSFORMERS_OFFLINE=1
19export HF_HUB_OFFLINE=1
20export HF_DATASETS_OFFLINE=1
21export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
22export NCCL_IB_DISABLE=0
23export NCCL_SOCKET_IFNAME="ib0,eno,eth"
24export WANDB_MODE=disabled
25
26################ 1. Distributed vars ############
27GPUS_PER_NODE=4
28NNODES=$SLURM_JOB_NUM_NODES
29NODE_RANK=$SLURM_NODEID
30MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n1)
31MASTER_PORT=29400
32RDZV_ID=$SLURM_JOB_ID
33
34################ 2. Launch ######################
35srun torchrun \
36 --nnodes $NNODES \
37 --nproc_per_node $GPUS_PER_NODE \
38 --rdzv_id $RDZV_ID \
39 --rdzv_backend c10d \
40 --rdzv_endpoint $MASTER_ADDR:$MASTER_PORT \
41 /path/to/nanotron/run_train.py \
42 --config-file smollm2/zagreus_350M_ita.yaml
Checkpoint Conversion to Hugging Face Format
1torchrun --nproc_per_node=1 -m examples.llama.convert_nanotron_to_hf \
2 --checkpoint_path=checkpoints/544000 \
3 --save_path=hf_checkpoints/544000 \
4 --tokenizer_name meta-llama/Llama-3.2-1B
Evaluation
Evaluation Commands
1lm-eval --model hf --model_args pretrained=<checkpoint> \
2 --tasks m_mmlu_it --num_fewshot 5 --device cuda:0 --batch_size 1
3
4lm-eval --model hf --model_args pretrained=<checkpoint> \
5 --tasks hellaswag_it,arc_it --device cuda:0 --batch_size 1
Checkpoint Progression
The table below tracks benchmark scores across training checkpoints, demonstrating steady model improvement throughout pre-training:
| Checkpoint | MMLU IT ↑ | HellaSwag IT ↑ | ARC IT ↑ | Average |
|---|
| v2-95k | 0.2529 | 0.3366 | 0.2652 | 0.2849 |
| v2-205k | 0.2628 | — | — | 0.2628 |
| v2-290k | 0.2428 | 0.3492 | 0.2335 | 0.2752 |
| v2-305k | 0.2598 | 0.3562 | 0.2652 | 0.2937 |
| v2-365k | 0.2566 | 0.3664 | 0.2712 | 0.2981 |
| v2-390k | 0.2556 | 0.3438 | 0.2498 | 0.2831 |
| v2-460k | 0.2540 | 0.3778 | 0.2549 | 0.2956 |
| v2-520k | 0.2540 | 0.3778 | 0.2549 | 0.2956 |
| v2-590k | 0.2547 | 0.3651 | 0.2455 | 0.2884 |
| v2-630k | 0.2562 | 0.3632 | 0.2643 | 0.2946 |
| v2-680k | 0.2538 | 0.3740 | 0.2592 | 0.2957 |
| v2-775k | 0.2535 | 0.3750 | 0.2583 | 0.2956 |
Evalita Benchmark
Evalita is a comprehensive Italian NLP evaluation suite benchmarking models across a wide range of linguistic tasks, from classification and extraction to generation and semantic understanding. Evaluation was conducted using the
evalita-mp task suite from
lm-evaluation-harness.
Evaluation Command
1lm_eval --model hf \
2 --model_args pretrained=mii-llm/zagreus-0.4B-ita \
3 --tasks evalita-mp \
4 --device cuda:0 \
5 --batch_size 1
Results
| Task | Metric | Score |
|---|
| Evalita-LLM (Overall) | acc | 0.3226 |
| Admission Test | acc | 0.2137 |
| FAQ | acc | 0.2681 |
| Hate Speech Detection | f1 | 0.6056 |
| Lexical Substitution | f1 | 0.0000 |
| NER | f1 | 0.1611 |
| Relation Extraction | f1 | 0.1244 |
| Sentiment Analysis | f1 | 0.3660 |
| Summarization (Fanpage) | rouge1 | 0.1947 |
| Text Entailment | acc | 0.5133 |
| Word in Context | f1 | 0.4697 |
Evalita results serve as a zero-shot baseline for the base model. For the comparison between the base model and the SFT variant, see the
Open-Zagreus-0.4B model card.
Usage
This is a base model — it performs causal language modelling (text completion) and is not instruction-tuned. It is best suited as a starting point for fine-tuning.
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
3
4model_id = "mii-llm/zagreus-0.4B-ita"
5
6tokenizer = AutoTokenizer.from_pretrained(model_id)
7model = AutoModelForCausalLM.from_pretrained(
8 model_id,
9 torch_dtype=torch.bfloat16,
10 device_map="auto"
11)
12
13# Base model: text completion, not instruction following
14prompt = "L'intelligenza artificiale è una disciplina che"
15
16inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
17
18output = model.generate(
19 **inputs,
20 max_new_tokens=200,
21 temperature=0.8,
22 do_sample=True,
23 repetition_penalty=1.1
24)
25
26print(tokenizer.decode(output[0], skip_special_tokens=True))
For instruction-following, use the post-trained variants:
- 🗣️ Nesso-0.4B-instruct — conversational and instruction following
- 🤖 Nesso-0.4B-agentic — function calling and agentic tasks
- 🔓 Open-Zagreus-0.4B — fully open-source SFT variant
Full Model Family
Base Models (Zagreus)
Post-trained Models (Nesso)
Citation
If you use this model in your research, please cite:
1@misc{zagreus2025,
2 title = {The Joy and Pain of Training an LLM from Scratch:
3 A Technical Report on the Zagreus and Nesso Model Families},
4 author = {mii-llm community},
5 year = {2025},
6 howpublished = {\url{https://github.com/mii-llm/zagreus-nesso-slm}},
7}
Acknowledgements
- Antonio Baldassarra (CEO, Seeweb) and Marco Cristofanilli (Head of AI, Seeweb) for commissioning and sponsoring the infrastructure
- The Hugging Face team for Nanotron, datatrove, FineWeb, FineWeb-2, and FinePDFs
- The mii-llm open-source community for contributions to multilingual evaluation harnesses and the Nanotron fork
License
Released under the Apache 2.0 license.
Made with ❤️ in Italy by
mii-llm
