Gemma 4 31B IT ZhTW CPT V2 10k
This repository stores three layer-wise LoRA adapters trained from google/gemma-4-31B-it with continued pretraining (CPT) on 10k Taiwan Traditional Chinese text examples.
Unlike the V1 instruction-tuning adapters, V2 uses plain text and causal language-modeling loss. Its goal is to test whether Taiwan-sourced dictionary and local text can change the model's natural preference for Taiwan terminology.
Dataset
- Mode: plain-text continued pretraining
- Format:
{"text": "..."}
- Training pool: 10,000 examples
- Independent validation set: 300 examples
- Per-run split: 9,800 train / 200 eval
- Total tokens per run: 322,800
- Average tokens per example: 32.28
- Maximum tokens per example: 265
| Source | Examples |
|---|
| MOE Concised Mandarin Chinese Dictionary definitions | 4,037 |
| MOE Concised Dictionary example sentences | 4,271 |
| MOE Dictionary of Chinese Idioms entries and examples | 1,558 |
| Taiwan travel text | 134 |
| Total | 10,000 |
The corpus is dominated by short dictionary-style text. It is useful for terminology adaptation, but it is not representative of complete Taiwan conversations or general instruction-following data.
Training Setup
- Base model:
google/gemma-4-31B-it
- Objective: causal LM loss over
text + eos
- Chat template: not used
- Quantization: 4-bit loading
- Precision: bf16
- Max sequence length: 1024
- Epochs: 1
- Per-device train batch size: 1
- Gradient accumulation steps: 8
- Learning rate: 2e-4
- LoRA rank: 16
- LoRA alpha: 32
- LoRA dropout: 0.05
- Target modules:
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- Hardware: one NVIDIA H100 PCIe
Adapters
| Folder | Strategy | LoRA layers | Train loss | Runtime |
|---|
strategy_a_tw_cpt_v2_10k/ | Overall critical layers | 12-19 | 4.6808 | 86.5 min |
strategy_b_tw_cpt_v2_10k/ | Critical + high Traditional/Simplified contrast layers | 8-10, 12-19, 58 | 3.3085 | 105.7 min |
strategy_c_tw_cpt_v2_10k/ | Critical + moderate layers | 1-24, 55-58 | 2.9838 | 119.3 min |
The strategies were derived from activation-based layer probing:
- Strategy A uses 8 overall-critical layers.
- Strategy B adds 4 layers with high Traditional/Simplified representation contrast.
- Strategy C uses a broader 28-layer candidate set.
Loading
Each strategy folder is a standalone PEFT adapter. Pass the repository ID and strategy folder separately:
1from peft import PeftModel
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4base_model_id = "google/gemma-4-31B-it"
5adapter_repo_id = "Esther125/gemma-4-31b-it-zhtw-cpt-v2-10k"
6adapter_subfolder = "strategy_b_tw_cpt_v2_10k"
7
8tokenizer = AutoTokenizer.from_pretrained(
9 adapter_repo_id,
10 subfolder=adapter_subfolder,
11)
12model = AutoModelForCausalLM.from_pretrained(
13 base_model_id,
14 device_map="auto",
15)
16model = PeftModel.from_pretrained(
17 model,
18 adapter_repo_id,
19 subfolder=adapter_subfolder,
20)
Evaluation
Taiwan terminology preference
A 200-prompt benchmark was built from 100 manually reviewed Taiwan/Mainland China terminology mappings. Each mapping has one controlled lexical prompt and one natural-generation prompt. Prompts do not request a specific regional variety.
| Model | Controlled TW preference | Natural TW preference | Overall TW preference | Trigger rate |
|---|
| Base | 90.6% | 88.3% | 89.5% | 81.0% |
| V2 Strategy A | 91.9% | 87.2% | 89.6% | 82.0% |
| V2 Strategy B | 94.4% | 92.5% | 93.5% | 84.5% |
| V2 Strategy C | 95.5% | 92.6% | 94.1% | 85.0% |
Strategies B and C improved preference for the benchmark's targeted Taiwan terms by 4.0 and 4.6 percentage points over the base model. Strategy B used 12 layers and finished within 0.6 percentage points of the 28-layer Strategy C.
This benchmark measures preference for 100 reviewed terminology mappings. It should not be interpreted as a complete measure of Taiwan Chinese generation quality.
Reasoning regression
Small ARC-Challenge, BBH Boolean Expressions, and GSM8K subsets were used as regression checks.
| Model | ARC | BBH | GSM8K | Macro accuracy |
|---|
| Base | 0.9333 | 1.0000 | 0.9333 | 0.9556 |
| V2 Strategy A | 0.9333 | 1.0000 | 0.8667 | 0.9333 |
| V2 Strategy B | 0.9333 | 1.0000 | 0.9333 | 0.9556 |
| V2 Strategy C | 0.9333 | 1.0000 | 0.8667 | 0.9333 |
These are small regression checks, not full benchmark results. Strategy B was the only V2 adapter that matched the base model's macro accuracy.
Recommended Checkpoint
Strategy B is the most balanced research checkpoint:
- 12 adapted layers instead of Strategy C's 28
- Taiwan terminology preference close to Strategy C
- Reasoning macro accuracy matched the base model in the small regression check
Limitations
- The corpus is small and mostly dictionary-based, with an average length of 32.28 tokens.
- The terminology benchmark covers 100 manually reviewed mappings and is not a general quality metric.
- Reasoning sets are small and designed for regression/strategy comparison.
- These adapters are research checkpoints, not production-ready models.
- Review the redistribution terms for the MOE dictionary and Taiwan travel sources before redistributing training data.
Related Repositories
- V1 10k instruction-tuning pilot: Esther125/gemma-4-31b-it-zhtw-lora-10k
- V1 50k instruction-tuning adapters: Esther125/gemma-4-31b-it-zhtw-lora-50k
- Source code and full experiment documentation: Esther125/gemma4-layer-probe-zhtw
