A conversational Welsh fine-tune of Google's Gemma 4 E4B, the small "effective 4B" model built for phones, laptops, Raspberry Pi class boards and anything else that has to work with no internet. The GGUF builds run in llama.cpp and LM Studio in about 5 to 9 GB of memory. Welsh should not need a data centre, and with this model it doesn't.
It speaks natural, everyday Welsh (correct mutations, consistent ti/chi register, idiomatic vocabulary) and keeps its English, so it answers in whichever language you use. This repository holds the full merged bf16 weights, ready for from_pretrained, plus the two LoRA adapters that training produced. Quantised GGUF files for edge deployment are in EryriLabs/gemma-4-e4b-cymraeg-v4-GGUF.
Trained on open Welsh data, most of it released by techiaith at Bangor University and by the Welsh public sector. Diolch o galon to everyone who put that data out under open licences; this model exists because they did.
Intended use
General-purpose Welsh and English conversation, drafting and question answering.
Commercial use is permitted for the base model and the named sources below.google/gemma-4-E4B-it is Apache-2.0. The continued-pretraining sources are all
commercially usable under their own licences (ODC-By 1.0, CC-BY-4.0, OGL v3, CC0),
and the fraud-awareness component of the instruction data is Crown copyright under
the Open Government Licence v3.0, which also permits commercial use. Reproduce the
attributions below where a licence requires it; see DATA-CARD-v4.md for the full
source list.
Not intended for: legal, medical or financial advice; or any setting where an
incorrect Welsh translation, or an incorrect fraud-related fact, carries
real-world risk without human review. Fraud-related facts (contact numbers,
organisation names, reporting routes) should be supplied through the system
prompt or retrieved context rather than relied on from the model's own memory ,
see README-stopthinkfraud.md.
Load with the class this checkpoint's config.architectures names, which is
Gemma4ForConditionalGeneration (model_type: gemma4): not the unified
Gemma4Unified... class used by the 12B. Loading E4B through the unified class
does not raise: transformers builds the wrong architecture, silently discards the
checkpoint tensors that do not fit it, and returns a partly-random model that
still produces confident, fluent-looking text. If you load it yourself, check the
loading info for missing or unexpected keys rather than trusting that it worked.
transformers >= 5.15 is required; earlier releases have no gemma4 model type.
Keep thinking off. Gemma 4 has a thought channel; leaving it enabled lets some
clients spend the whole token budget reasoning and return an empty answer. This
model was trained and evaluated with thinking off.
Recommended sampling (LM Studio / llama.cpp):
setting
value
temperature
0.3
top_p
0.9
min_p
0.05
repeat_penalty
1.05
Training summary
Two-stage LoRA training: continued pretraining on raw Welsh, then supervised
fine-tuning on Welsh instructions.
Stage 1: continued pretraining (CPT)
corpus built
85,269 documents / 78,002,622 tokens (see DATA-CARD-v4.md for per-source counts and corpus_report.json for drop counts)
corpus trained on
43,835,392 tokens: a uniform random subset of the above, drawn after a global shuffle so every source is represented in proportion
method
LoRA, rank-stabilised scaling (rsLoRA)
LoRA
r=128, alpha=32, dropout 0
targets
q, k, v, o, gate, up, down projections
embeddings
embed_tokens trained in full via modules_to_save (lm_head is tied to it in this architecture, so the two are one matrix). 950M of 6.9B parameters trainable: 279M LoRA + 671M embedding
learning rate
1e-4 (LoRA weights), 1e-5 (embedding matrix), separate optimiser groups
schedule
cosine, 2% warmup, 668 optimiser steps
sequence length
2048, raw-only blocks, labels = input_ids
batch
32 x 1 (no gradient accumulation)
precision
bf16, 4-bit base (QLoRA)
epochs
1
The trained subset is smaller than the built corpus because the run was capped by
a fixed compute budget, not because the remainder was unusable. Sizing was done
from a measured 20-step benchmark on the training GPU (6,100 tokens/s sustained,
48.2 GB peak VRAM) rather than an estimate, and the corpus was shuffled before
the cap was applied so the subset is representative rather than a truncation of
the source order. The full 78M-token corpus and its build report ship with this
repository.
Stage 2: supervised fine-tuning (SFT)
data
19,443 Welsh instruction rows (~4.8M tokens) mixed with a halved raw replay slice: 3,586 documents / 2,500,523 tokens sampled from 7,202 / 5,000,299: against a 370-row validation split. See DATA-CARD-v4.md and DATA-CARD-v4-sft.md
mix ratio
~2:1 instruction to raw. The raw share was reduced after the sibling 12B v3 run, whose stage 2 was raw-heavy (14.3M raw against 4.2M SFT), was judged to have regressed on Welsh with repetition collapses and content-free brevity: the signature of a mix that teaches continuation rather than answering
base
the stage-1 CPT-merged checkpoint, not the stock model
method
LoRA (embedding matrices not trained in this stage)
LoRA
r=64, alpha=64
learning rate
8e-5
schedule
cosine, 3% warmup
sequence length
2048
epochs
1
batching
homogeneous batches (raw blocks and chat rows never share a batch); final 20% of batches are instruction-only, so the low-learning-rate end of the schedule is spent entirely on answering
Both adapters are shipped in adapters/cpt/ and adapters/sft/. The merged bf16
weights at the root of this repository incorporate both, applied in training
order (CPT, then SFT).
Evaluation
Blind LLM-as-judge scoring (Opus), identical prompts and settings for every arm
(nf4, thinking off, greedy, 400 max tokens, no system prompt). The judge is not
a native speaker; structural findings (language choice, loops, checkable facts)
are reliable, fine idiom judgements less so: native review welcome.
Single-turn, 40 Welsh prompts
model
replies in the requested language
fluency /5
grammar /5
task /5
stock gemma-4-E4B-it
47.5 %
2.46
2.38
1.45
v3 (SFT only)
97.5 %
3.87
3.59
2.33
v4 (CPT 43.8M tokens + SFT)
100 %
3.68
3.76
2.70
Pairwise v4 vs v3 (blind): 23 wins, 13 ties, 4 losses. English is fully
retained (fluency 5.0, task 4.95: best in the family). Invented Welsh words
roughly halved versus v3; the corpus-installed mutations (i Gaerdydd,
i Fangor, dy fore di) are v3's exact failures now correct.
Multi-turn (14 fixed turns, history kept, sampling as recommended above)
model
conversation score /5
stock
1.97
v3
3.35
v4
3.99
Quantisation: Q6_K scored highest of the three quants under llama.cpp
(fluency 4.42, grammar 4.36); Q4_K_M showed no measurable damage; all three
quants: 0 corrupt tokens on the corruption-prone subset.
Known limitations: occasional repetition loops on long open-ended answers
(mitigated by repeat_penalty); factual recall (dates, statistics) is weak,
supply facts via the system prompt or retrieval; when given long retrieved
context it can over-quote it. Fraud-safety facts must not be relied on from
memory. Full judgement report and per-response scores:
eval/judgement_e4b_v4.md, eval/scores_e4b_v4.jsonl.
A head-to-head against BangorAI/Mistral-7B-Cymraeg-Welsh-v2 on the same
protocol is in the section below.
Raw outputs and training logs are in eval/. GGUF corruption counts per
quantisation are in the GGUF repo's MANIFEST.json.
Head-to-head with BangorAI Mistral-7B-Cymraeg-Welsh-v2
BangorAI/Mistral-7B-Cymraeg-Welsh-v2
is the strongest open Welsh model we know of and the only open comparison in the
7B class, so it is the bar this model is measured against. It is a genuinely good
Welsh model, and everything below should be read as a measurement against a high
bar rather than as a criticism of it. Diolch to BangorAI for publishing it.
It is not a like-for-like comparison, and it is not meant to be:
this model (v4)
BangorAI 7B v2
parameters
6.9B total, ~4B effective per token
7B dense
Welsh continued pretraining
43.8M tokens, 1 pass
MADLAD Welsh, 2 epochs, on the order of 412M token-passes
ratio
1x
~9.4x the Welsh tokens, ~1.75x the active parameters
BangorAI was run with its own documented Welsh system prompt, which is its
intended interface, in the single-turn arm and in the multi-turn sys arm. The
multi-turn nosys arm is the properly matched comparison. Sampling,
quantisation, prompts and token limits were identical throughout.
Single-turn, 40 Welsh prompts plus 20 English prompts
arm
replies in the requested language
Welsh fluency /5
Welsh grammar /5
task /5
collapses
English task /5
v4
40/40
3.68
3.76
2.70
5
4.95
BangorAI 7B v2
37/40
4.85
4.65
3.35
0
4.20
Pairwise on the same 40 prompts: BangorAI 23 wins, 13 ties, v4 4 wins. That is
the mirror image of v4 against v3. BangorAI's Welsh essentially never breaks:
zero degeneration loops and zero corrupt word-forms anywhere. v4's four wins are
the three prompts that ask for English output, which BangorAI answers in Welsh,
plus one conversational prompt. v4 leads on task in translation (3.50 against
2.00) and assistant chat (3.43 against 2.86); BangorAI leads everywhere else.
Multi-turn, 14 fixed turns with history kept
arm
fluency
grammar
relevance
conversational
memory
mean
v4 / sys
4.07
4.14
3.21
3.36
3.69
3.70
v4 / nosys
4.29
4.14
3.79
3.86
3.85
3.99
BangorAI / sys
4.79
4.93
3.36
3.00
2.92
3.80
BangorAI / nosys
4.64
4.93
3.21
2.93
3.00
3.74
Per-turn wins, ties and losses:
condition
v4
tie
BangorAI
sys vs sys (each with its own intended prompt)
5
2
7
nosys vs nosys (matched)
5
4
5
combined, 28 turns
10
6
12
v4 wins the turns that hold a conversation together: the opening greeting, the
summary, the close (both v4 conditions produce an eleven-turn callback,
"Mwynha dy daith i Aberystwyth", which no BangorAI condition does), and above
all turn 11, the request to switch to English. v4 switches cleanly and
translates faithfully in both conditions; BangorAI stays in Welsh in both, and
under its own system prompt it states in Welsh that it can only help in English.
BangorAI wins the turns that reward content and polish: the tips, the facts, the
joke, the fraud advice and the poem.
Verdict in three lines. BangorAI wins overall and wins Welsh polish
decisively: it is more fluent, more grammatical and never degenerates, and if you
want the best open Welsh output and can run a 7B, use it. The conversation is far
closer, and in the matched condition v4 takes it on the arm mean (3.99 against
3.74) and draws on turns, so roughly 4B effective parameters trained on a tenth of
the Welsh fights a 7B to a draw over fourteen turns. What the extra scale and data
buy is surface quality and degeneration resistance; what they do not buy is
memory, register mirroring or switching language on request, which is why a small
bilingual model still has a place on edge devices.
Caveats. The judge is an LLM (Opus) and not a native Welsh speaker;
structural findings are reliable, fine idiom judgements are not, and a non-native
judge is least reliable exactly where BangorAI wins most, so the fluency gap is
more likely understated than overstated. Sizes and Welsh token counts are not
matched. BangorAI ran through its own system-prompt interface, so only the
multi-turn nosys comparison is matched on prompt text. The rubric was written
for a friendly conversational model, which favours v4 on register and warmth.
Sample sizes are small. Full report and per-item scores:
eval/judgement_v4_vs_bangorai.md and eval/scores_v4_vs_bangorai.jsonl in the
base repository.
Limitations
Welsh costs more tokens per word than English for the Gemma 4 tokeniser, which
reduces the effective context length available for Welsh text relative to
English.
Some of the instruction data was generated or adapted by an automated pipeline
rather than written by a native speaker; treat Welsh fluency as unverified for
anything that will be published verbatim.
Fraud-related facts (numbers, organisation names, reporting routes) can be
misremembered or go stale. Supply them through the system prompt or retrieved
context rather than relying on the model's memory: see
README-stopthinkfraud.md.
GGUF quantisation introduces measurable degradation at lower bit-widths; read
the corruption counts in the GGUF repo's MANIFEST.json before choosing a
quantisation below Q6_K.
Stage 1 was a single epoch over 43.8M tokens, not the full 78M-token corpus
that was built (the run had a fixed compute budget). That is enough to move the
embedding table and the adapters, but it is a modest amount of continued
pretraining and its effect should not be overstated: compare against the v3
arm in eval/ rather than assuming an improvement.
No tool-calling, no vision use, and no guarantee of factual accuracy.
Repository contents
path
contents
repo root
merged bf16 weights, ready for from_pretrained
adapters/cpt/
stage 1 continued-pretraining LoRA adapter
adapters/sft/
stage 2 supervised fine-tuning LoRA adapter
eval/
raw evaluation outputs and training logs
DATA-CARD-v4.md
full corpus description, per-source counts, filter chain, contamination control
DATA-CARD-v4-sft.md
stage-2 instruction-set composition
corpus_report.json
machine-readable corpus statistics and drop counts
eval/licence_exclusion_report.json
the 447 instruction rows excluded pending licence clarification
README-stopthinkfraud.md
provenance and licence of the Stop! Think Fraud Welsh fraud-awareness data used in training
Attributions
Base model google/gemma-4-E4B-it (trained from the unsloth/gemma-4-e4b-it
mirror), Apache-2.0.
techiaith/banc-trawsgrifiadau-bangor
(CC0) was not pulled directly: it is gated on the Hub and the build was
refused access. It was targeted at ~1.3M tokens (1.7% of the corpus) and was not
re-attempted, because passing the gate means accepting the dataset's terms.
Bangor transcription material is still present: 3,321 of the 20,560 documents
carried forward from the V3 corpus carry source: banc: so the register is
represented but was not topped up. corpus_report.json records the failure.
Per-source row counts, drop counts and the exact filter chain are in
DATA-CARD-v4.md.
447 rows were built and then removed before tokenisation: 427 from
sft_train.jsonl and 20 from sft_val.jsonl, all of them from the
macsen_intent_json and macsen_intent_conv sources. The underlying techiaith
dataset states no licence, and this release claims Apache-2.0 with commercial use
permitted: a row whose licence cannot be evidenced cannot sit inside that claim.
They are excluded pending licence clarification rather than shipped with a
caveat. Counts are in eval/licence_exclusion_report.json; the SFT set actually
trained on is 19,443 train / 370 validation rows.
Licence pass-through
Where a source licence requires attribution, that requirement passes through to
users of this model:
OGL v3 sources: reproduce "Contains public sector information licensed under
the Open Government Licence v3.0."
