This is the flagship release of CantoneseLLM v2. It is a 30.5B-parameter
mixture-of-experts model (3.3B activated) based on Qwen3-30B-A3B that reasons and answers in Hong Kong
Cantonese.
It is the final checkpoint of a five-stage pipeline: continual pre-training →
chat-vector merge → supervised fine-tuning → DPO → two-stage RLVR. The full
development history, including the stages that failed and why, is documented in
the technical report.
Open models prompted in Cantonese answer in Cantonese but reason in another
language. Here is the same translated GSM8K probe, under a system prompt that
explicitly asks for Hong Kong Cantonese.
Chat-vector merged checkpoint — opens in Cantonese, then slides into Written
Chinese and ends up discussing the output language instead of thinking in it:
…現在，用香港廣東話回應。…在回應中，要用口語廣東話。
(…Now, respond in Hong Kong Cantonese. … In the response, colloquial Cantonese should be used.)
This model — reasons in Cantonese end to end, with the Cantonese classifier
隻 zek3 and colloquial forms such as 剩低 zing6 dai1:
(First, I have to break down the steps of this arithmetic problem. It says Janet's ducks lay 16 eggs a day, she eats 3 in the morning, 4 go into baking muffins, and what is left is sold at the farmers' market.)
Both reach the correct answer, 18 dollars. What changes is where the reasoning
happens.
The mechanism is a multiplicative language and script term in the RLVR
reward — task_reward × language_multiplier × format_factor — so a
wrong-language trace cannot buy its way back by being correct.
Trace length over eight fixed probes (mean tokens between the reasoning tags):
Checkpoint
Mean CoT tokens
CoT-to-answer ratio
Qwen3-30B-A3B-Thinking-2507 (official)
1,042
6.25
Chat-vector merged
1,442
5.85
This model
155
1.43
The official and merged checkpoints are predominantly Written Chinese across
those probes. This model's traces are Cantonese — and short, which is a real
limitation rather than a design goal (see below).
Read this table carefully. SFT cost 12.46 points; RLVR recovered 8.99 of
them, 88% of what was lost after the merge. The model finishes 1.20 points
below the merged checkpoint it started from — while carrying the Cantonese
reasoning behaviour that the merged checkpoint never had, plus translation,
data-judging and Cantonese lexical capability that multiple-choice questions do
not measure.
If you select on benchmark average alone you will pick the merged checkpoint and
get a model that reasons in Written Chinese. That trade is the subject of the
paper.
Artefacts released with this work
Every post-training stage on this card is reproducible from public components.
The CPT corpus itself is not released, but its two largest public constituents
are.
The six NeMo-Gym environments used in RLVR stage 2 — math_with_judge_lang, stem_mcqa, code_gen_lang, workplace_assistant_lang, instruction_following_lang, structured_outputs_lang. Each implements the multiplicative language reward, task_reward × language_multiplier × format_factor. Targets NeMo-Gym 0.3.0rc0; MIT licensed
The RLVR stage-2 prompt corpus. NVIDIA's Nemotron-3-Nano RL blend translated into Cantonese and Hong Kong Written Chinese, kept as a three-way parallel corpus (en/, yue/, zh-hk/) so the same question can be compared across languages
The Cantonese subset of the above, filtered with CantoneseDetect and globally deduplicated to 477,298 unique documents
The three-way parallel structure of the RL corpus is what makes the
cross-language pass-rate gap reported in the paper measurable: for the
translated environments, prompt language is the only variable.
The intermediate checkpoint this model was built from: CPT + chat-vector merge, before SFT, DPO and RLVR. Scores higher on HKCanto-Eval but reasons in Written Chinese — it is the baseline the paper measures against, not a replacement for this model
Add --tensor-parallel-size N to shard across N GPUs, and
--reasoning-parser qwen3 if you want the reasoning block returned separately
as reasoning_content rather than inline in content (parser names vary by
vLLM version).
OpenAI-compatible API
The served endpoint speaks the OpenAI protocol, so the official client works
unchanged:
Sampling: temperature 0.6, top-p 0.95 — the settings used for every
evaluation reported above and in the paper. Avoid greedy decoding.
System prompt: the Cantonese system prompt above was used throughout
training and evaluation. Behaviour with other system prompts, or with none, is
not characterised.
This is a thinking-only model. Unlike the 8B, its SFT mixture was entirely
chain-of-thought, so it has not been trained to answer without a reasoning
block. Do not expect /no_think to behave as it does in the stock Qwen3 hybrid
models.
Training pipeline
Stage
What it installed
Compute
Continual pre-training
Hong Kong knowledge and Cantonese lexis. 784M tokens, 530 steps, LR 1.5×10⁻⁵, on 64 TPU v6e chips (Google TRC) via MaxText
Translation, data curation and LLM-as-a-judge behaviour. 74,865 rows / 177.4M tokens, 1,575 steps, NeMo-RL (Megatron)
107 GPU-hours
DPO
Repaired the reasoning-block format regression. 14,679 preference pairs, 435 steps
115 GPU-hours
RLVR stage 1
Output format and CoT language, on a multilingual GSM8K blend (50% Cantonese / 25% Written Chinese / 25% English). 150 steps, LoRA r64
33 GPU-hours
RLVR stage 2
Broadened the reward across six environments. 300 steps, LoRA r64, DAPO-style clipping with expert parallelism
1,223 GPU-hours
Stage 2 took four attempts; the first three (≈775 GPU-hours) were abandoned to
entropy collapse and importance-sampling problems. Attempt 4 is this checkpoint.
The failure analysis is in the paper — it is one of the more useful parts.
Released weights are the final step of stage 2, not a best-validation
checkpoint.
Risks & Limitations
Short reasoning traces. 155 tokens on average. No long original Cantonese
reasoning traces existed at the scale SFT needed; human-written or
human-verified traces amounted to 131 rows. Most Cantonese CoT in training
was machine-translated from English or Simplified Chinese. The model reasons
in the right language but does not reason at length.
No Hong Kong grounding in the reasoning data. The SFT mixture contained no
content grounded in Hong Kong entities or current events — that knowledge
comes from CPT only.
Benchmark scores are underestimates of knowledge. DPO's pair-selection
weighted longer responses with no reward for instruction compliance, so the
model sometimes ignores "answer with the letter only". Multiple-choice parsers
that read a bare leading option letter will under-score it. Use a
marker-anchored extractor.
Language steering may over-fire toward Cantonese. At the intermediate
stage-1 checkpoint the model produced Cantonese in 87.1% of rollouts on
mathematics even when explicitly asked for Written Chinese. Stage 2 trained
all three languages with the multiplier, but test this if you need reliable
zh-hk output.
One run per stage. Severe compute constraints meant no hyperparameter
sweep in any post-training stage.
Preference data were model-judged, not human-annotated, and are bounded by
the judge's own Cantonese ability.
Long context is inherited and untested. Training sequence lengths were
8,192 (SFT/DPO) and 16,384 (RLVR stage 2). Behaviour beyond that is whatever
the Qwen3 base provides.
Standard LLM caveats apply: it will hallucinate, and it has not been safety-
tuned beyond what the chat vector carried over.
Citation
bibtex
1@misc{cantonesellm_v2,
2 title={CantoneseLLM v2: Reasoning in a Low-Resource Language},
3 author={Tsz Chung Cheng and Chung Shing Cheng and Chaak Ming Lau and Cheuk Hei Chong},
4 year={2026},
5 eprint={2609.06970},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2609.06970},
9}
Acknowledgements
Continual pre-training (CPT) was carried out on Cloud TPUs (Tensor Processing Units) from Google's TPU Research Cloud with MaxText. Post-training was carried out on computer resources offered under the category of General Projects by Research Institute for Information Technology, Kyushu University. Usage fee and cost of data-curation costs with proprietary APIs were covered by Votee AI
