⚠️ Preliminary results — 1 seed, variance not controlled. Every number below is a single training run evaluated once (50 prompts × 200 tokens, fixed seed). Directions are consistent, magnitudes are not validated. Treat as a lab notebook, not a benchmark.

Focal is a 15M-parameter French language model trained from scratch (no 🤗 Transformers, no pretrained weights) on a single GTX 1080 Ti. It is the in-domain member of a three-model family: a looped transformer with learned per-token adaptive halting using an absolute entropy threshold.

Identity

Architecture: decoder-only, LLaMA-style (RoPE, RMSNorm, SwiGLU, QK-Norm, Flash/SDPA attention), looped R=4 (effective depth 32) + per-token adaptive halting: a token stops looping once its output entropy falls below a fixed threshold (1.0 nats). Zero added parameters — the entropy comes from logits already computed. Halting is active during training, so the model learns to be good at whatever depth each token stops.
Size: 15.01M parameters · n_embd 256 / n_layer 8 / n_head 4 · context 768.
Language: French, from scratch.
Tokenizer: custom ByteLevel BPE, 32k vocab (bpe_tokenizer_32k.json).
Data: French corpus (AI-rewritten Wikipedia + filtered web via RDTextract), ~425M tokens total. These runs used 20% of the train split, 2 epochs (fast research protocol).

What Focal is good at

Focal is the in-domain model: learned halting acts as a training-time regularizer that sharpens quality on prompts close to the training distribution.

Best in-domain coherence: 44.1 (vs 35.2 baseline) — the highest of the four models.
Best prompt overlap: 0.176 (stays most on-topic).
Trade-off: it is the weakest out-of-domain (coherence 29.1). On far-off prompts, the sharpening becomes a liability.
Note: on real text the halting rarely fires at inference (most tokens run full depth) — the gain is a training effect, not an inference speedup.

Robust evaluation (50×200, 1 seed)

Same table on all three model cards, so the trade-off is visible everywhere. Auto = held-out corpus prompts (in-domain); Fixed = generic hand-written prompts (out-of-domain).

Model	Val PPL ↓	Coherence auto ↑	Coherence fixed ↑	Invented names auto ↓	Prompt overlap auto ↑
Baseline (vanilla)	31.2	35.2	40.7	0.137	0.139
Cadence (looped R=4)	28.9	39.3	32.5	0.121	0.147
Focal (absolute halt)	29.1	44.1	29.1	0.103	0.176
Nomade (percentile halt)	31.0	36.3	41.8	0.094	0.126

No model wins everything — that is the result. Recurrence helps in-domain and hurts out-of-domain; the halting variants trade one for the other. Factuality is not improved by any of them (15M capacity ceiling).

Lineage (this is not novel)

ACT — Adaptive Computation Time, Graves 2016 (per-token variable depth) · arXiv:1603.08983
Universal Transformer — Dehghani et al. 2018 (the looped/weight-shared ancestor) · arXiv:1807.03819
Recurrent-Depth Transformer — Geiping et al. 2025 (Huginn-3.5B; the canonical RDT this family follows) · arXiv:2502.05171
CALM — Confident Adaptive Language Modeling, Schuster et al. 2022 (confidence early-exit across layers) · arXiv:2207.07061
LoopViT — Shu et al. 2026 (parameter-free entropy exit + weight-tied loop) · arXiv:2602.02156

Direct basis — the two from-scratch looped studies this work reproduces in French: Kohli et al. 2026 (arXiv:2604.07822, COLM) and Chen 2026 (arXiv:2603.21676).

Focal is a variant of LoopViT for autoregressive language, with halting learned during training (per token, entropy criterion). No claim of novelty — the ingredients are all prior work.

Limitations

1 seed, no variance control — magnitudes may reorder with more seeds.
20% of data, 2 epochs — a short research protocol, not a full training run.
15M capacity — hallucinations are expected; the model learns form and coherence, not facts. It will not reliably tell you the capital of France.
Absolute entropy threshold does not transfer across model sizes (a bigger model is more confident) — for scale-up, use the percentile variant (Nomade).
No instruction tuning — a pure completion model.

✍️ Write-up (Article 3): Teaching a 15M French LLM to think deeper — Hugging Face blog
🔁 Cadence (looped reference): RDTvlokip/Cadence-15M-fr
🧭 Nomade (out-of-domain, size-invariant halting): RDTvlokip/Nomade-15M-fr

Usage

This model uses custom from-scratch code (not 🤗 Transformers). The code ships with the repo (model/, utils/), so a snapshot download is self-contained. Requires torch, huggingface_hub, tokenizers.

python

import sys, torch
from huggingface_hub import snapshot_download

repo = snapshot_download("RDTvlokip/Focal-15M-fr")  # code + weights + tokenizer
sys.path.insert(0, repo)
from model.gpt2 import GPT2
from utils.tokenizer import GPT2Tokenizer

# The absolute-halting config travels inside the checkpoint — nothing extra to set.
model = GPT2.from_pretrained(f"{repo}/best_model.pt", device="cuda"); model.eval()
tok = GPT2Tokenizer(f"{repo}/bpe_tokenizer_32k.json")

ids = tok.encode("Au Moyen Âge, les chevaliers", add_special_tokens=True)
if ids and ids[-1] == tok.eos_token_id:
    ids = ids[:-1]                         # drop trailing <eos> (keeps it on-topic)
out = model.generate(
    torch.tensor([ids], device="cuda"),
    max_length=80, temperature=0.8, top_k=40, top_p=0.9,
    repetition_penalty=1.3, eos_token_id=tok.eos_token_id,
)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

Théo CHARLET — TSSR Graduate, AI/ML · Creator of AG-BPE · rdtvlokip.fr

RDTvlokip/Focal-15M-fr

Focal-15M-fr 🎯