gemma-4-12b-cymraeg-v6
The flagship conversational Welsh model in the EryriLabs Gemma 4 line. It is
Gemma 4 12B instruction tuned, continued pretrained on a large Welsh corpus and
then supervised fine tuned for conversation, so it answers in natural Welsh
rather than translating from English as it goes.
What separates this release from the ones before it is the amount of Welsh the
model was actually immersed in. Continued pretraining ran in two blocks over
complementary halves of the corpus, and between them they covered
307M of the 336M token corpus. Block B started from the merged output of Block A and drew only
on rows Block A had never seen, so the two blocks add coverage rather than
repeat it.
This repository holds the supervised fine tuning adapter and the evaluation
record. It does not hold merged bf16 weights, and it does not hold the two
continued pretraining adapters: those live in their own repositories, named
below, and copying them here a third time would serve nobody.
Conversational benchmark results
Method first: 100 fresh conversational Welsh prompts were authored for this test and machine checked for zero overlap against every earlier prompt set and against all training data of every model involved. Three models answered under their own documented settings. Scoring was blind (shuffled slots, deterministic seed) on a rubric fixed before any answer existed: helpfulness, correctness and naturalness of the Welsh, register, and fitness of length for a chat turn. The judge is a language model, not a native speaker; its per prompt scores, its uncertainty list, and the full prompt sets are published in eval/ so anyone can re mark the papers.
On that benchmark this model won 66 prompts to 33 with 1 tie against
BangorAI/Mistral-7B-Cymraeg-Welsh-v2 (win rate 66.7 percent, 95 percent CI 56.9 to 75.2, p = 0.0012), and won a separate 14 turn dialogue test 4.34 to 3.74 with the strongest cross turn memory of any model tested. In the same set of experiments BangorAI's model won a 20 prompt mixed category exam 13 to 7 at each model's shipping settings and remains the stronger long form prose writer, with the most polished Welsh sentence by sentence (language quality 4.52 vs 4.18). The difference is behavioural: in the conversational hundred, BangorAI's replies included assistant style boilerplate 25 times, refused a reasonable request 17 times and missed the ti/chi register 15 times, against 0, 1 and 5 for this model.
BangorAI's model is the strongest other open Welsh model available and the only 7B class comparison; this work also trains on open data released by techiaith at Bangor University and by the Welsh public sector. Diolch o galon i bawb.
| flag counts per 100 conversational replies | this model | BangorAI 7B |
|---|
| assistant boilerplate | 0 | 25 |
| refusals of reasonable requests | 1 | 17 |
| ti/chi register misses | 5 | 15 |
| replies rated conversation ready | 67 | 44 |
Lineage: google/gemma-4-12b-it, continued pretraining on about 307 million tokens of open Welsh text (the full MADLAD-400 Welsh split plus the techiaith corpora; embeddings trained in the first stage), then conversational supervised fine tuning. See DATA-CARD files and cpt_progress.json for exact coverage.
What is in this repository
| Path | What it is |
|---|
adapters/sft/ | Stage 2 adapter: supervised fine tuning for conversation, applied on top of the merged continued pretraining chain |
adapters/cpt/POINTER.md | Where the two continued pretraining adapters live, and the order they must be applied in |
eval/ | Every evaluation output produced for this release, as JSONL, plus the training logs |
DATA-CARD-v6.md | The continued pretraining corpus, its sources and their licences |
DATA-CARD-v6-sft.md | The stage 2 conversational data |
How the merged weights were produced
The bf16 merged checkpoint is not uploaded. The weights come from applying three
LoRA adapters to the base model, in this order:
- CPT Block A, trained on the first slice of the corpus, with a trained
embed_tokens table.
- CPT Block B, trained on top of base plus Block A and restricted to rows Block A had never seen, with the embedding table frozen.
- The stage 2 supervised fine tuning adapter, included in this repository under
adapters/sft/.
The order mattered: Block B was trained on top of base plus Block A, so applying
it directly to the stock base would have produced a different model.
CPT Block A and CPT Block B were kept as private checkpoint repositories and
were never published, so the merge above cannot be reproduced from public
sources. This section describes how the weights were built, not a set of steps a
reader can carry out.
corpus_manifest.json in this repository describes the composition of the
continued pretraining corpus. The corpus itself is not published.
Quantised GGUF builds of the merged model are published separately at
EryriLabs/gemma-4-12b-cymraeg-v6-GGUF.
Intended use
Welsh language conversation, question answering and general assistance, with
English understood as a second language. The model is a general purpose
assistant and is not a substitute for professional advice in any field.
Evaluation
Blind LLM as judge (Opus), identical prompts and settings for every arm. The judge is not a native speaker: structural findings are reliable, fine style judgements less so. Full report and per response scores are in eval/.
Welsh, 40 prompts:
| model | fluency | grammar | task | conversation-ready replies | loops |
|---|
| v2 (5.9M raw tokens) | 4.00 | 3.85 | 3.20 | 20/40 | 4 |
| v5 (71M CPT) | 3.90 | 3.85 | 3.23 | 22/40 | 4 |
| v6 greedy (307M CPT) | 4.25 | 4.15 | 3.27 | 23/40 | 2 |
| v6 sampled (recommended) | 4.35 | 4.10 | 3.17 | 22/40 | 0 |
| BangorAI Mistral 7B v2 | 4.85 | 4.65 | 3.35 | 14/40 | 0 |
Head to head with BangorAI/Mistral-7B-Cymraeg-Welsh-v2, the strongest other open Welsh model and the only 7B class comparison (diolch to BangorAI and to techiaith at Bangor University, whose open data trained much of this model): single turn pairwise 15 wins, 5 ties, 20 losses (BangorAI keeps the single turn edge; the gap has narrowed from 14-21 for v2 to 16-18 for v6 greedy). Multi turn conversation, 14 fixed turns with history: v6 4.34 vs BangorAI 3.74, per turn 8-2-4, with cross turn memory the largest gain (3.92 to 4.46). English is fully retained (4.90 fluency, 20/20 good English).
v6 vs v5 pairwise: 16 wins, 16 ties, 8 losses. The additional immersion bought coherence and fact repair more than sentence elegance.
Known limitations: long form answers past ~150 tokens can leak English or loop under greedy decoding (cap generation at ~160 tokens and use the sampling below, which removes every observed catastrophic failure); rare unprovoked refusals (2 of 80 responses); it performs mutations far better than it explains them; numeric facts should be grounded via the system prompt.
Recommended settings: temperature 0.3, top_p 0.9, min_p 0.05, repetition_penalty 1.1, max_new_tokens 160, thinking off.
Training
Stage 1, continued pretraining, ran as two sequential blocks over complementary
halves of the corpus, together covering 307M of the 336M token corpus.
- Block A: LoRA r 128 on the attention and MLP projections, plus a trained
embed_tokens table through modules_to_save, on the first slice of the
corpus.
- Block B: LoRA r 128 on the same projections, embeddings frozen, trained on top
of base plus Block A and restricted to rows Block A had never seen. Freezing
the embedding table is why Block A's most expensive tensor survives the chain
intact rather than being partly relearned on a smaller sample.
Stage 2, supervised fine tuning: LoRA r 64, alpha 64, dropout
0, on the seven linear projections only, with no embedding training. Learning
rate 5e-5 with a cosine schedule and a 0.03 warmup ratio, sequence length
2048, one epoch. The conversational rows are mixed with a sampled replay of
the stage 1 corpus so that stage 2 does not undo stage 1, and the last
20 percent of the training order is conversation only, so the low
learning rate end of the schedule is spent entirely on answering rather than on
continuing text.
Data and attribution
The full source list with token counts is in the data card. The corpus is built
from openly licensed material only:
- MADLAD-400, Welsh portion, under ODC-BY.
- Welsh Government material via llyw.cymru, under the Open Government Licence.
- Cofnod y Cynulliad and Senedd record material, under the Open Government
Licence.
- Cardiff University translation memory, under CC-BY.
- Banc Trawsgrifiadau Bangor, under CC0.
The techiaith collections are gratefully acknowledged. Licence terms travel with
the data: anything requiring attribution is attributed here and in the data card.
Licence
Apache 2.0, matching the base model google/gemma-4-12b-it. The corpus licences above apply
to the training data, not to the model weights.
Limitations
- Welsh is a lower resource language and the model will still make mistakes,
including confident ones. Check anything that matters.
- Mutations, register and dialect are uneven. Formal written Welsh is stronger
than colloquial speech.
- The model can produce English when pushed hard in English, especially without
a Welsh system prompt. The multi turn arms measure exactly this.
- No safety tuning beyond what the base model carries.
