AfriqueQwen3.5-0.8B-50Langs is part of the AfriqueLLM suite, a collection of open language models adapted to 50 African languages through continued pre-training (CPT) on ~35.5B tokens. This model is based on Qwen/Qwen3.5-0.8B-Base and has been specifically adapted for improved performance on African languages while maintaining strong capabilities in high-resource languages.

This compact variant follows the same extended code, math, and 50-language continued-pretraining recipe as AfriqueQwen3.5-4B-50Langs.

Key Features

Type: Causal Language Model (Base/Pre-trained)
Base Model: Qwen 3.5 0.8B
Parameters: 0.8B
Context Length: 262,144 tokens (native)
Training Tokens: ~35.5B tokens of carefully curated multilingual data

Supported Languages

AfriqueQwen3.5-0.8B-50Langs has been adapted for the following 50 African languages:

Language	Code	Family	Script
Afrikaans	afr_Latn	Germanic	Latin
Swahili	swh_Latn	Bantu	Latin
Moroccan Arabic	ary_Arab	Semitic	Arabic
Somali	som_Latn	Cushitic	Latin
Amharic	amh_Ethi	Semitic	Ethiopic
Egyptian Arabic	arz_Arab	Semitic	Arabic
Hausa	hau_Latn	Chadic	Latin
Kinyarwanda	kin_Latn	Bantu	Latin
Zulu	zul_Latn	Bantu	Latin
Igbo	ibo_Latn	Volta-Niger	Latin
Plateau Malagasy	plt_Latn	Austronesian	Latin
Xhosa	xho_Latn	Bantu	Latin
Shona	sna_Latn	Bantu	Latin
Yoruba	yor_Latn	Volta-Niger	Latin
Nyanja	nya_Latn	Bantu	Latin
Southern Sotho	sot_Latn	Bantu	Latin
Tigrinya	tir_Ethi	Semitic	Ethiopic
Tunisian Arabic	aeb_Arab	Semitic	Arabic
Oromo	gaz_Latn	Cushitic	Latin
Tswana	tsn_Latn	Bantu	Latin
Rundi	run_Latn	Bantu	Latin
Ganda	lug_Latn	Bantu	Latin
Tsonga	tso_Latn	Bantu	Latin
Lingala	lin_Latn	Bantu	Latin
Ewe	ewe_Latn	Kwa	Latin
Wolof	wol_Latn	Senegambian	Latin
Sango	sag_Latn	Creole	Latin
Akan/Twi	aka_Latn / twi_Latn	Kwa	Latin
Kabiye	kbp_Latn	Gur	Latin
Bambara	bam_Latn	Mande	Latin
Northern Sotho	nso_Latn	Bantu	Latin
Fon	fon_Latn	Kwa	Latin
Swati	ssw_Latn	Bantu	Latin
Central Atlas Tamazight	tzm_Tfng	Berber	Tifinagh
Kabyle	kab_Latn	Berber	Latin
Kabuverdianu	kea_Latn	Creole	Latin
N'Ko	nqo_Nkoo	Mande	N'Ko
Mossi	mos_Latn	Gur	Latin
Kimbundu	kmb_Latn	Bantu	Latin
Kanuri	knc_Arab / knc_Latn	Saharan	Arabic/Latin
Dyula	dyu_Latn	Mande	Latin
Tamasheq	taq_Latn	Berber	Latin
Southwestern Dinka	dik_Latn	Nilotic	Latin
Luo	luo_Latn	Nilotic	Latin
Nigerian Fulfulde	fuv_Latn	Senegambian	Latin
Bemba	bem_Latn	Bantu	Latin
Kikuyu	kik_Latn	Bantu	Latin
Kamba	kam_Latn	Bantu	Latin
Kikongo	kon_Latn	Bantu	Latin
Luba-Kasai	lua_Latn	Bantu	Latin

High-resource languages used for catastrophic forgetting mitigation: English, French, Portuguese, Arabic

Training Data

Our training corpus combines multiple high-quality sources:

African Monolingual Data (~22.8B tokens): FineWeb2, WURA, and MADLAD-400
Code (~1B tokens): CornStack-Python for reasoning capabilities
Mathematics (~1B tokens): FineMath-4+ for mathematical understanding
Synthetic Data (~324M tokens): GPT-4.1 translated domain-specific content across 10 domains
Additional Language Expansion (~1.5B tokens beyond the extended code/math recipe): remaining African language data, upsampled 5x for broader language coverage.

We use UniMax sampling to create a balanced distribution, capping high-resource languages at approximately 1B tokens and upsampling lower-resource languages for up to five epochs.

Quickstart

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "McGill-NLP/AfriqueQwen3.5-0.8B-50Langs"

# Load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# Prepare the model input
prompt = "Bawo ni o ṣe n ṣe?"  # Yoruba: "How are you doing?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate text
generated_ids = model.generate(
    **inputs,
    max_new_tokens=100,
)
output = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
print(output)

Deployment

For deployment, you can use vllm or sglang to create an OpenAI-compatible API endpoint:

vLLM:

vllm serve McGill-NLP/AfriqueQwen3.5-0.8B-50Langs

SGLang:

python -m sglang.launch_server --model-path McGill-NLP/AfriqueQwen3.5-0.8B-50Langs

Training Details

Hyperparameters

Learning Rate: 5e-5 (with warmup and cosine decay)
Context Length: 16,384 tokens
Optimizer: AdamW
Precision: BF16 mixed precision

Infrastructure

Training was conducted using the LLaMA-Factory framework on up to 64 NVIDIA H100 GPUs with:

DeepSpeed ZeRO-1/ZeRO-2
Flash Attention 3
Sequence packing
Liger Kernel optimizations

Evaluation

All AfriqueLLM models are evaluated on multiple multilingual benchmarks. FLORES is reported only in the English-to-target direction (eng->xxx):

Model	AfriMGSM	AfriMMLU	AfriXNLI	Belebele	FLORES (eng->xxx)	INJONG	SIB-200	Overall	Δ (Δ %)
Gemma3-4B	10.24	33.89	37.76	45.79	35.36	55.52	63.59	40.31
AfriqueGemma-4B	14.86	36.73	39.62	50.52	54.95	69.28	69.21	47.88	+7.6 (18.8%)
Gemma3-12B	25.21	48.76	44.01	68.84	44.09	73.53	79.17	54.80
AfriqueGemma-12B	32.14	49.47	44.60	68.65	65.04	76.79	75.08	58.82	+4.0 (7.3%)
Qwen3-4B	8.26	33.84	37.12	41.50	20.16	21.69	57.88	31.49
AfriqueQwen-4B	33.09	43.04	44.88	63.62	59.82	65.34	74.77	54.94	+23.4 (74.4%)
Qwen3.5-0.8B	3.51	28.72	33.52	31.24	14.80	3.66	39.07	22.07
AfriqueQwen3.5-0.8B-50Langs	8.37	28.29	33.96	32.71	49.85	24.55	51.18	32.70	+10.63 (+48.1%)
Qwen3.5-4B	20.79	38.63	40.36	55.82	32.06	59.43	74.96	46.01
AfriqueQwen3.5-4B	30.47	43.66	41.05	66.01	63.55	75.46	79.66	57.12	+11.1 (24.2%)
AfriqueQwen3.5-4B-ExtendedCM	34.17	45.26	41.94	66.45	63.76	75.97	80.52	58.30	+1.2 (2.1%)
AfriqueQwen3.5-4B-50Langs	34.06	45.23	41.79	66.83	64.56	75.82	79.83	58.30	+0.0 (0.0%)
Qwen3.5-9B	31.61	44.00	44.14	63.87	40.80	71.06	81.12	53.80
AfriqueQwen3.5-9B-50Langs	40.47	49.76	45.93	72.01	65.00	78.53	82.42	62.02	+8.22 (+15.3%)
Qwen3-8B	11.22	36.56	38.24	44.63	21.13	29.47	53.06	33.47
AfriqueQwen-8B	39.68	46.91	45.99	68.46	62.18	73.36	77.00	59.08	+25.6 (76.5%)
Qwen3-14B	16.60	39.66	43.22	50.74	23.61	41.80	66.29	40.27
AfriqueQwen-14B	45.01	52.22	49.01	74.63	63.77	77.80	82.63	63.58	+23.3 (57.9%)
Llama3.1-8B	8.14	32.27	37.90	40.95	26.69	41.37	59.99	35.33
AfriqueLlama-8B	17.51	36.57	37.39	50.51	63.60	71.17	69.14	49.41	+14.1 (39.9%)
Lugha-Llama-8B-wura	9.46	37.00	39.24	47.86	49.90	62.30	75.81	45.94
Gemma3-27B	35.37	55.47	46.85	74.81	48.41	79.70	84.34	60.71

Additional-Language Evaluation

This table averages only evaluated African languages outside the first 20-language CPT set: Ewe, Lingala, Ganda, Twi, and Wolof. Benchmark cells average the available languages for that benchmark; FLORES is English-to-target only (eng->xxx).

Model	AfriMGSM	AfriMMLU	AfriXNLI	Belebele	FLORES (eng->xxx)	INJONG	SIB-200	Avg	Δ (Δ %)
Qwen3.5-0.8B	2.24	26.68	33.08	28.50	15.52	5.60	35.50	21.02
AfriqueQwen3.5-0.8B-50Langs	3.71	25.35	33.01	28.69	27.64	14.10	42.23	24.96	+3.94 (+18.8%)
Qwen3.5-4B	8.30	32.35	34.30	36.90	22.20	33.27	58.96	32.33
AfriqueQwen3.5-4B	7.52	31.17	33.56	35.82	26.28	33.10	56.66	32.02	-0.31 (-1.0%)
AfriqueQwen3.5-4B-ExtendedCM	8.13	32.72	33.74	36.76	24.06	32.41	56.00	31.97	-0.05 (-0.2%)
AfriqueQwen3.5-4B-50Langs	21.07	37.65	36.31	51.56	56.33	61.37	75.75	48.58	+16.61 (+52.0%)
Qwen3.5-9B	14.42	34.94	36.10	42.04	20.14	45.50	66.72	37.12
AfriqueQwen3.5-9B-50Langs	25.30	40.06	38.27	55.58	58.93	68.60	79.52	52.32	+15.20 (+40.9%)

Model Variants

AfriqueQwen-14B - Qwen-based 14B model (flagship)
AfriqueQwen-8B - Qwen-based 8B model
AfriqueQwen-4B - Qwen-based 4B model
AfriqueQwen3.5-9B-50Langs - Qwen 3.5-based 9B model with 50-language coverage (Qwen 3.5 flagship)
AfriqueQwen3.5-4B - Qwen 3.5-based 4B model
AfriqueQwen3.5-4B-ExtendedCM - Qwen 3.5-based 4B model with extended continued pre-training
AfriqueQwen3.5-4B-50Langs - Qwen 3.5-based 4B model with 50-language coverage
AfriqueGemma-4B - Gemma-based 4B model
AfriqueGemma-12B - Gemma-based 12B model
AfriqueLlama-8B - Llama-based 8B model

Citation

If you find our work helpful, please cite:

bibtex

@misc{yu2026afriquellmdatamixingmodel,
      title={AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages}, 
      author={Hao Yu and Tianyi Xu and Michael A. Hedderich and Wassim Hamidouche and Syed Waqas Zamir and David Ifeoluwa Adelani},
      year={2026},
      eprint={2601.06395},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2601.06395}, 
}

License

This model is released under the CC BY 4.0 License. Please review the license terms before use.

Acknowledgments

We thank the creators of the base models, datasets and compute resources that made this work possible, including Mila, Compute Canada, Microsoft, the FineWeb team, WURA, MADLAD-400 and etc..