Hypa-Gemma4 E2B (hypaai/Hypa-Gemma4-E2B-v1) is a LoRA-merged fine-tune of Google DeepMind's Gemma 4 E2B-it, produced by Hypa Intelligence. It is the first model released in our open research line on adapting modern open foundation models for low-resource and underrepresented languages, with a deliberate focus on retaining the base model's tool-aware and agentic prompting structure.
This release covers seventeen languages: English, French, Spanish, and fourteen languages of Nigeria. Several of the smaller languages in this set (including Annang, Eggon, Idoma, Igala, Nupe, and Urhobo) have not been formally represented in large-scale fine-tuning corpora before, or had no settled ISO-style language tag at the time we needed one.
The model is intended for translation, language detection, dictionary-style explanation, and general multilingual instruction-following. It inherits Gemma 4's native chat template, system / user / model role structure, and dedicated formatting for thinking and tool use.
Property
Value
Base model
google/gemma-4-E2B-it (2.3B effective, 5.1B with PLE)
Method
LoRA (r=1024, α=1024) via Unsloth + QLoRA, then merged to 16-bit
Trainable parameters
1.62B / 6.74B (24.04%)
Training data
15.9M examples across 11 concatenated sub-datasets
Compute
1× NVIDIA H200 SXM, 7.41 days
Languages
17
Context window
128K (inherited from base model)
License
Apache 2.0
Quick Start
python
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
34model_id ="hypaai/Hypa-Gemma4-E2B-v1"56tokenizer = AutoTokenizer.from_pretrained(model_id)7model = AutoModelForCausalLM.from_pretrained(8 model_id,9 dtype=torch.bfloat16,10 device_map="auto",11)1213messages =[14{"role":"system","content":"You are an expert Igbo translator."},15{"role":"user","content":"Translate to Igbo: Good morning, how are you today?"},16]1718inputs = tokenizer.apply_chat_template(19 messages,20 tokenize=True,21 return_tensors="pt",22 add_generation_prompt=True,23).to(model.device)2425outputs = model.generate(26 inputs,27 max_new_tokens=256,28 temperature=1.0,29 top_p=0.95,30 top_k=64,31 do_sample=True,32)3334print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
For thinking mode, pass enable_thinking=True to apply_chat_template. For tool-use payloads, use the standard Gemma 4 messages-with-tools format.
Languages Covered
Code
Language
Code
Language
en
English
ibb
Ibibio
ann
Annang
idm
Idoma
efi
Efik
igl
Igala
ebi
Ebira
ig
Igbo
ego
Eggon
nup
Nupe
es
Spanish
pg
Pidgin
fr
French
tiv
Tiv
ha
Hausa
urh
Urhobo
yo
Yoruba
Some of the smaller languages in this set required custom or non-standard tags because no widely-adopted machine-readable code existed at the time of training. Where ISO 639-3 codes were available, we used them; where they were not, we documented our internal codes in the data release so downstream users can reproduce splits.
Training Data
Training data comprises 15.9 million examples assembled from a large multilingual text mixture combining internal Hypa datasets and public instruction-style corpora. The overall training mixture included dictionary-style data, translation data, language detection data, synthetic instruction data, and function-calling data. Each contributing a different signal:
A public 10k subset of the training data is released as hypaai/Hypa-Text-10k. Additional sub-datasets are progressively being released under the hypaai organization.
Prompt Formatting
Every example was formatted using Gemma 4's native chat template, with explicit system, user, and model roles and dedicated control tokens for thinking (<|think|>, <|channel>thought ... <channel|>) and tool use. Loss was computed only on assistant turns via train_on_responses_only with instruction_part="<|turn>user\n" and response_part="<|turn>model\n".
Training was performed using Unsloth, which provides day-zero Gemma 4 support and patches the shared-KV-cache interaction with gradient checkpointing that affects naive QLoRA setups in vanilla transformers.
Evaluation and Recommendations
Training metrics
Final training loss: 0.41 (smooth monotonic decay)
Best evaluation loss: 3.128 at step 40,000
Final evaluation loss: 3.577
Honest note on overfitting
Evaluation loss bottomed out at step 40,000 (≈16% of total training) and rose steadily for the remaining ~209,000 steps, ending ~14% above its best value. The training loss continued to decrease throughout. This is a clear overfitting signature: the model memorized the training distribution beyond the point where additional optimization improved generalization.
For downstream use, we recommend the LoRA checkpoint at step 40,000 rather than the final merged 16-bit weights published in this repository. The intermediate checkpoint is available at hypaai/Hypa-Gemma4-E2B-v1-LoRAs. The merged 16-bit weights in this repo represent the end-of-training state and are useful as a reference but are not the strongest checkpoint by evaluation loss. However, this training wasn't as good.
The full discussion of why this happened and what we will change for v2 is in the public write-up. Headline planned fixes: lower LoRA rank, load_best_model_at_end=True with eval_loss as the selection metric, language-held-out evaluation splits, and a lower step count.
Qualitative observations
Internal qualitative review on translation tasks shows minor improvements over the base Gemma 4 E2B for every language in the set, with the largest deltas on the smallest languages (Annang, Efik, Ibibio), where the base model was effectively unusable. Quantitative chrF++, BLEU, and BLEURT results across language pairs will follow in a separate evaluation post.
Intended Use
Direct use cases:
Translation between English / French / Spanish and the fourteen covered low-resource languages
Language detection across all seventeen languages
Dictionary-style lexical lookup and explanation
Multilingual instruction-following on dialogue tasks
Tool-aware / function-calling-style prompting (inheriting the base model's structure)
Downstream use:
Suitable as a starting point for further fine-tuning on more specialized tasks within the supported languages
Suitable for adapter stacking (e.g., domain-specific LoRA on top)
Suitable for on-device deployment when quantized (E2B at 4-bit fits on mid-range mobile hardware)
Out-of-Scope and Limitations
Not safety-tuned for sensitive domains. This model has not undergone RLHF or DPO post-training. It should not be used unsupervised for medical, legal, financial, or psychological-counseling applications.
Quality varies by language. The smallest languages in the set are underrepresented even within our training mix and the resulting model output should be reviewed by native speakers before being used in production.
Overfitting on the final checkpoint. As noted above, the merged 16-bit weights in this repository correspond to the end of training, not the best evaluation checkpoint. For applications that prioritize generalization, use the step-40,000 LoRA from the companion repository.
Vision and audio components are frozen. This is a text-only fine-tune. The base model's image and audio capabilities are preserved at the underlying weight level but were not exercised during training and have not been validated for our target languages.
Tokenization quality. Gemma 4's 256K-vocabulary tokenizer fragments low-resource languages less aggressively than smaller-vocabulary tokenizers, but the smallest languages in this release still tokenize at higher cost per character than English. This is the gap we expect future iterations to close.
Coverage is finite. The seventeen languages in this release are the start, not the end. Many other underrepresented languages are not yet supported and may produce unreliable output.
Bias, Risks, and Limitations
This model inherits the biases and limitations of its base model (Google Gemma 4) and adds the biases of its fine-tuning corpus, which is weighted toward dictionary, religious-parallel, and CommonVoice text. Religious-parallel text in particular is a known cause of register and content bias in low-resource translation models. Users deploying this model in customer-facing applications should evaluate output for cultural appropriateness in their specific use case and language.
The model is not intended to make decisions affecting people's rights, health, finances, or wellbeing. Like all language models, it can produce confident-sounding output that is incorrect, particularly on the smallest languages where training data was thinnest.
If you use Hypa-Gemma4 E2B or any of the related work, please cite:
bibtex
1@misc{hypaai2026hypagemma4e2b,
2 title = {Hypa-Gemma4 E2B: A Multilingual Fine-Tune of Gemma 4 for Underrepresented Languages},
3 author = {{Hypa Intelligence}},
4 year = {2026},
5 publisher = {Hugging Face},
6 howpublished = {\url{https://huggingface.co/hypaai/Hypa-Gemma4-E2B-v1}},
7 note = {Apache 2.0 License. Blog: \url{https://hypaintelligence.com/updates/tuning-gemma-4-for-multilingual-and-tool-aware-language-understanding}}
8}
License
Released under the Apache License 2.0, inheriting the license of the Gemma 4 base model. Free to use, modify, and redistribute for both research and commercial purposes, with no monthly active user caps and no attribution friction.
Acknowledgments
Google DeepMind for releasing Gemma 4 under Apache 2.0 and for its 140+ language pretraining commitment.
Unsloth for day-zero Gemma 4 support and for handling the KV-sharing interaction with gradient checkpointing.
Runpod for reliable H200 infrastructure.
The language communities, speakers, and reviewers whose texts, voices, and feedback grounded this work and keep it honest.
