A 442M-parameter French language model using a Recurrent-Depth Transformer architecture, fully designed and trained from scratch by Eric Houzelle.

Every component — architecture, training pipeline, and inference engine — was written in PyTorch without relying on any pre-trained weights or third-party model code. Trained end-to-end on a single NVIDIA L40S GPU.

Key Facts


Parameters	442M
Architecture	Recurrent-Depth Transformer (RDT)
Language	French
Tokenizer	CamemBERT (`camembert-base`, 32k vocab)
Context window	512 tokens
Chat format	ChatML
Training hardware	1x NVIDIA L40S
SFT val_loss	1.56
License	Apache 2.0

Quick Start

pip install transformers torch safetensors sentencepiece

Conversational Mode (ChatML)

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Klovis-ai/Klovis-442M-french-130426"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

prompt = (
    "<|system|>\n"
    "Tu es un assistant intelligent. Réponds directement et précisément.<|end|>\n"
    "<|user|>\n"
    "Que peut-on visiter à Paris ?<|end|>\n"
    "<|assistant|>\n"
)

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    inputs["input_ids"],
    max_new_tokens=200,
    temperature=0.5,
    top_p=0.85,
    top_k=50,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Special Tokens

Token	Role
`<\|system\|>`	Start of system message
`<\|user\|>`	Start of user message
`<\|assistant\|>`	Start of assistant response
`<\|end\|>`	End of turn

Architecture — Recurrent-Depth Transformer

Klovis uses a Recurrent-Depth Transformer (inspired by OpenMythos/Parcae, Prairie et al. 2026), where a single Transformer block is applied iteratively:

Input → [Prelude: 4 layers] → [Shared Block x 8 steps] → [Coda: 4 layers] → Output

Component	Detail
Embedding dim	1536
Prelude layers	4 (unique)
Coda layers	4 (unique)
Recurrent steps	8
Query heads	24
KV heads (GQA)	6
FFN hidden dim	8192
FFN activation	SwiGLU
Normalization	RMSNorm (pre-norm)
Position encoding	RoPE
LTI Injection	Linear Time-Invariant state coupling
ACT Halting	Adaptive Computation Time (per-position early exit)
Depth LoRA	Rank-32 adapters per recurrent step
Weight tying	Input embeddings <-> output projection

Why Recurrent-Depth?

The RDT architecture achieves 442M effective parameters with significantly fewer unique parameters by reusing a single Transformer block across 8 depth steps. This enables:

Deeper reasoning through iterative refinement
Dynamic compute via ACT (positions that are "easy" halt early)
Step specialization via Depth LoRA adapters

Training

Phase 1 — Pre-training


Data	French texts (Wikipedia FR, FineWeb-2, CulturaX, C4)
Optimizer	AdamW (beta1=0.9, beta2=0.95, wd=0.1)
Scheduler	Linear warmup -> cosine decay
Precision	Mixed precision (bfloat16)
Block size	512 tokens

Phase 2 — Supervised Fine-Tuning (SFT)

Fine-tuned on curated French conversational datasets across 25 epochs, with prompt masking (only assistant tokens contribute to the loss).


Epochs	25
Learning rate	2e-5
Effective batch size	128 (16 x 8 gradient accumulation)
Final val_loss	1.56
Chat format	ChatML

SFT Datasets

Dataset	Samples
kurakurai/luth-sft (multiple subsets)	~300k
angeluriot/french_instruct	20k
Houzeric/everyday-conversations	Custom
Houzeric/physics-FR-qa-dataset	5k
le-leadboard/MATH_LVL5_fr	1.4k

Example Outputs

Here are real outputs from the model (temperature=0.2, top_p=0.7, top_k=30):

User: Bonjour

Klovis: J'espère que vous allez bien ! Je suis ravi de partager mon expérience avec vous aujourd'hui. Si vous avez des questions spécifiques ou si vous souhaitez approfondir mes connaissances dans ce domaine, n'hésitez pas à me le faire savoir !

User: Que puis-je visiter à Paris ?

Klovis: La Tour Eiffel est une structure emblématique située dans le cœur historique de Paris. Construite pour l'Exposition universelle de 1889, elle offre une vue imprenable sur la ville. Cette avenue pavée offre une expérience unique de shopping et d'histoire, avec ses nombreux cafés, restaurants et boutiques de luxe qui témoignent de son riche passé. Les visiteurs peuvent profiter d'une promenade matinale ou d'un dîner gastronomique dans un restaurant étoilé Michelin.

User: Explique-moi ce qu'est la gravité.

Klovis: La gravité est une force qui agit sur tout objet ou chose, comme l'air ou l'eau dans son ensemble. Les effets de cette accélération peuvent varier considérablement selon les conditions atmosphériques. En résumé, la gravité est un concept fondamental en physique moderne, représentant la gravité comme un phénomène dynamique influencé par des facteurs tels que les mouvements tectoniques, les forces gravitationnelles et les champs électromagnétiques.

[!NOTE] The model produces fluent, well-structured French with rich vocabulary. Factual accuracy is limited (as expected for a 442M model trained from scratch) — responses may contain hallucinations or approximations.

What to Expect

With 442M parameters, Klovis produces:

Grammatically correct, fluent French
Well-structured responses (numbered lists, bold headers, paragraphs)
Relevant vocabulary across science, culture, history
Coherent conversational interactions

Known limitations:

Factual responses may be incorrect or fabricated (hallucinations)
Mathematical reasoning is limited
Context limited to 512 tokens
French only

This model demonstrates what a single developer can achieve with a novel architecture at small scale. It is not intended to replace larger models for production use.

License

Apache 2.0 — https://www.apache.org/licenses/LICENSE-2.0

Designed and trained by Eric Houzelle.