Qwen3-32B's BPE tokenizer over-segments common Korean endings and particles (어미/조사) into 2-4 sub-tokens. This model adds them as single tokens, trained via QLoRA to be natively used during generation.

What was done

	Before	After
`했잖아`	`했` + `잖` + `아` (3 tokens)	`했잖아` (1 token)
`봤는데`	`봤` + `는데` (2 tokens)	`봤는데` (1 token)
`죄송하지만`	`죄` + `송` + `하지만` (3 tokens)	`죄송하지만` (1 token)
Vocab size	151,669	155,351 (+3,682)

The 3,682 tokens were extracted from HyperCLOVA's Korean-optimized vocabulary — specifically endings (어미) and particles (조사) that Qwen's BPE consistently fragments.

Training

Method: QLoRA (r=64, alpha=128) on Colab A100
Key technique: Old embedding freeze — gradient hook zeros out gradients for the original 151K token embeddings, forcing the optimizer to only update the 3,682 new token rows
Data: ~77K Korean samples filtered for high new-token density (≥5 target tokens per sample), sourced from KoAlpaca, alpaca-gpt4-korean, KULLM-v2
Epochs: 1 (with high-density data + freeze, convergence is fast)
New token initialization: Mean pooling of constituent sub-token embeddings

Training curve

Step	Loss	Accuracy
50	1.649	65.6%
500	1.186	70.9%
1000	1.140	71.8%

Results

New token adoption rate: 92.9% — when the model generates text containing a string that matches a new token, it uses the single new token ID 92.9% of the time (vs. falling back to the old fragmented sub-tokens).

Prompt	Adoption	New tokens used
어제 친구를 만났는데 걔가 갑자기...	4/4 = 100%	`났지만`, `달라고`, `했지만`, `갔습니다`
솔직히 그건 좀 아닌 것 같거든?	1/1 = 100%	`좋아하는`
한국의 경제 성장에 대해 설명해주세요	1/1 = 100%	`때문`
이거 진짜 맛있거든? 너도 한번 먹어봐	3/3 = 100%	`보세요`, `있으며`, `하다고`
Write a Python function...	0/0 = N/A	(no Korean tokens expected)

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "2264K/Qwen3-32B-KoTokenizer"

# NF4 quantization (fits in 24GB VRAM)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

# Verify new tokens work
print(tokenizer.encode("했잖아", add_special_tokens=False))
# [155305]  ← single token (was 3 tokens before)

# Generate
messages = [{"role": "user", "content": "어제 친구를 만났는데 걔가 갑자기 이상한 얘기를 하더라고."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9)

print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Important notes

This is a merged model (not an adapter). Load it directly like any HuggingFace model.
The tokenizer is included. No need to load the base Qwen3-32B tokenizer separately.
The model's generation style is unchanged from Qwen3-32B — this modification only affects tokenization efficiency, not the model's personality or capabilities.
English and code generation are unaffected (0 new tokens in English outputs, as expected).

Files

model-*.safetensors — merged model weights (bf16)
tokenizer.json, tokenizer_config.json — expanded tokenizer
token_expansion_metadata.json — metadata for all 3,682 added tokens (token strings, IDs, source sub-token IDs used for mean pooling init)

License

Apache 2.0 (same as Qwen3-32B)