GLM-4-32B-0414 Korean-Culture QLoRA — v0.1 (pilot)
This is a small pilot adapter, published for transparency — not recommended for production use.
A QLoRA (LoRA r=16) adapter that fine-tunes
zai-org/GLM-4-32B-0414 on 1,750 rows of Korean data, targeting the Korean-culture knowledge gap of Chinese-origin base models. Part of an openly documented attempt to climb the Korean K-AI leaderboard
honestly — full recipe, scripts, and failure log:
github.com/jwson-automation/korean-glm-merge.
Training data (1,750 rows, ~1.2M tokens)
| Slice | Rows | Source |
|---|
| Culture/knowledge Q&A | 1,093 | Generated from Korean Wikipedia (220 verified articles) by Qwen2.5-32B-Instruct (Apache-2.0), self-verified at temperature 0, closed-book format |
| Human-written instructions | 357 | CohereLabs/aya_dataset Korean subset (Apache-2.0) |
| Spoken dialogue | 300 | 국립국어원 모두의 말뭉치 spoken corpus (subsampled; the corpus itself is redistribution-restricted and is not included here — only weights) |
Decontamination: all training rows were filtered against a 49k-item blocklist built from KMMLU (all subjects), CLIcK, HAE-RAE Bench 1.1, and KoBEST (3-layer filter: 13-word shingles, short-item substring, 5-word Jaccard). Note: this catches verbatim/near-verbatim reuse only, not semantic paraphrases.
Honest evaluation (pilot-scale: lm-eval-harness, --limit 100 per subtask, bf16, 0-shot)
⚠️ These are relative comparison numbers on a 100-item-per-subtask sample, not full benchmark scores.
| base | +this adapter | Δ |
|---|
| CLIcK (aggregate) | 67.0 | 68.0 | +1.0 (within noise) |
| CLIcK culture | 70.9 | 73.7 | +2.8 |
| CLIcK language | 58.0 | 55.0 | −3.0 |
| HAE-RAE Bench | 72.2 | 70.8 | −1.4 |
The interesting pattern: all 8 culture subtasks moved up or stayed flat (tradition 86→90, society 74→78, kpop 75.6→87.8, politics 79.8→82.1, …) while all 3 language subtasks moved down — i.e. the model learned roughly what the data covered (culture) and paid a small tax where it had no data (grammar/usage). Aggregate is a wash.
Known limitations
- Quiz-format overfit: open-ended questions often get answered in a "정답: …" quiz style.
- Language/grammar regression (no language-track data in this pilot).
- Teacher and verifier were the same model (Qwen2.5-32B), so teacher hallucinations can survive verification; a manual audit estimated ~75% good / ~10% bad / ~15% marginal items.
- Trained in 4-bit (QLoRA); evaluated with the adapter applied to the bf16 base.
Usage
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
3
4base = AutoModelForCausalLM.from_pretrained("zai-org/GLM-4-32B-0414", dtype="bfloat16", device_map="auto")
5model = PeftModel.from_pretrained(base, "puchuneko/GLM-4-32B-0414-Korean-Culture-QLoRA")
6tokenizer = AutoTokenizer.from_pretrained("puchuneko/GLM-4-32B-0414-Korean-Culture-QLoRA")
Lineage & credits
- Base: zai-org/GLM-4-32B-0414 (MIT)
- Teacher: Qwen/Qwen2.5-32B-Instruct (Apache-2.0)
- Related: puchuneko/GLM-4-32B-0414-Z1-SLERP (round-1 merge from the same project)
No benchmark training data was used. No SOTA claims. v0.2 plan: scale culture Q&A 5–10k with a cross-model verifier, add a dedicated grammar/usage track, and test continued pretraining on the source corpus before SFT.
