Qwen3.5-35B-A3B-Kazakh
Qwen3.5-35B-A3B-Kazakh is a Kazakh adaptation of
Qwen/Qwen3.5-35B-A3B. It
extends the tokenizer with an additional Kazakh vocabulary for more efficient tokenization, is
continued-pretrained on Kazakh text, and is then
re-aligned to the official instruct model. This makes the model more efficient on Kazakh text and adds knowledge, while preserving the capabilities of the base model.
Methodology
This model is produced in
two stages on top of our Kazakh continued-pretraining model
issai/Qwen3.5-35B-A3B-Base-Kazakh, the official
Qwen/Qwen3.5-35B-A3B-Base continued-pretrained on Kazakh text with a
+16K Kazakh-token vocabulary extension. No additional post-training (SFT, RLHF, or RLVR) is used.
1. Chat-vector merge, transferring instruct alignment
We transfer the instruction-following / chat alignment of the official post-trained model
using the
chat vector method
(
Huang et al., 2024). Let
θ_inst: weights of the official post-trained Qwen/Qwen3.5-35B-A3Bθ_base: weights of the official base Qwen/Qwen3.5-35B-A3B-Baseθ_cpt: weights of our Kazakh base model issai/Qwen3.5-35B-A3B-Base-Kazakh
The chat vector is the task vector of post-training,
$$\tau_\text{chat} = \theta_\text{inst} - \theta_\text{base},$$
and it is added to our Kazakh continued-pretraining model:
$$\theta_\text{merged} = \theta_\text{cpt} + \tau_\text{chat} = \theta_\text{cpt} + (\theta_\text{inst} - \theta_\text{base}),$$
applied per parameter tensor, by name. Because our tokenizer is vocabulary-extended, the
input-embedding and output (lm_head) matrices are merged row-wise: the shared
original-vocabulary rows receive the chat vector, while the 16K new Kazakh-token rows are
kept from θ_cpt (the official models have no counterpart for them). embed_tokens and lm_head are untied and are merged independently. This model is a Mixture-of-Experts; the official checkpoints store the experts fused (experts.gate_up_proj, experts.down_proj), so our per-expert weights are fused to that layout before the merge, and the router / shared-expert weights are merged by name like any other tensor.
Special-token pinning. The rows of the format-control special tokens
<think>, </think>, <|im_start|>, <|im_end|> and <|endoftext|> (in both the
embedding and lm_head) are set to the official post-trained model's exact values
instead of the merged values. These few rows drive the chat template's control flow (emitting </think> to close the
reasoning block and <|im_end|> to end the turn). The raw chat vector otherwise detunes
them and breaks this template logic, causing the model to loop and never terminate. Pinning them restores reliable termination
while preserving the Kazakh knowledge acquired during continued pretraining.
2. Vision re-integration, restoring multimodality
Because our continued pretraining was performed on text only, with the vision branch removed beforehand, the chat-vector merge operates on the language model and produces a text-only model. To
restore image and video understanding, the
vision encoder and multimodal projector from
the official
Qwen/Qwen3.5-35B-A3B are re-attached to this Kazakh-adapted,
instruct-aligned language backbone, and the architecture is set back to
Qwen3_5MoeForConditionalGeneration. The result is a multimodal model that largely preserves the official model's vision
capabilities while its language model handles Kazakh (and English) with instruct-style
alignment and correct
<think>...</think> reasoning.
Evaluation
We evaluate
Qwen3.5-35B-A3B-Kazakh against its base model
Qwen3.5-35B-A3B on Kazakh and English benchmarks under identical settings (thinking mode enabled):
temperature = 1.0top_p = 0.95top_k = 20presence_penalty = 1.5max_tokens = 81920
Scores are accuracy (%).
⚠️ Note on reasoning-heavy benchmarks. The custom model tends to reason more briefly (shorter chains-of-thought) than the original Qwen model, and this is most visible on the hardest problems. The largest regressions therefore appear on the hardest reasoning benchmarks in our evaluation set, such as PolyMath (5.8 pts lower) and GPQA-Diamond (4.5 pts lower), while on easier benchmarks the effect is minimal and the model preserves the original Qwen model's capabilities.
⚠️ Note on vision benchmarks. The merged model was not explicitly trained on vision tasks, which accounts for some degradation on the vision benchmarks. These results can be further improved with dedicated vision-language finetuning.
Kazakh
The Kazakh benchmarks (most of which are available in the ISSAI
Qolda suite:
language,
vision) use Kazakh questions and answer choices.
KazMMLU and
KazCulture are native Kazakh benchmarks; the rest are Kazakh adaptations of the standard benchmarks.
Text
| Benchmark | Qwen3.5‑35B‑A3B | Qwen3.5‑35B‑A3B-Kazakh |
|---|
| KazMMLU | 82.6 | 84.0 | | KazCulture | 60.3 | 68.7 | | MMLU | 87.2 | 84.4 | | MMLU-Pro | 79.3 | 76.1 | | GPQA | 70.9 | 69.8 | | ARC-Easy | 98.1 | 97.5 | | ARC-Challenge | 96.3 | 95.5 | | GSM8K | 92.7 | 89.3 | | Belebele | 89.4 | 88.9 | | MMLU-Redux | 88.4 | 85.9 |
|
Vision
| Benchmark | Qwen3.5‑35B‑A3B | Qwen3.5‑35B‑A3B-Kazakh |
|---|
| AI2D | 85.7 | 85.2 | | MMStar | 76.0 | 75.2 | | RealWorldQA | 77.5 | 78.6 | | MathVista | 83.3 | 82.8 | | OCRBench | 42.4 | 34.0 | | MathVision | 81.4 | 78.8 | | MMBench | 91.5 | 91.7 |
|
English
Text
| Benchmark | Qwen3.5‑35B‑A3B | Qwen3.5‑35B‑A3B-Kazakh |
|---|
| MMLU | 91.2 | 89.3 | | MMLU-Pro | 85.5 | 82.7 | | GPQA-Diamond | 86.9 | 82.3 | | GPQA-Main | 79.7 | 76.6 | | ARC-Easy | 99.3 | 97.6 | | ARC-Challenge | 98.0 | 95.7 | | GSM8K | 97.0 | 94.3 | | Belebele | 97.4 | 97.7 | | MMLU-Redux | 92.1 | 89.9 |
|
Vision
| Benchmark | Qwen3.5‑35B‑A3B | Qwen3.5‑35B‑A3B-Kazakh |
|---|
| AI2D | 90.8 | 90.6 | | MMStar | 78.8 | 79.3 | | RealWorldQA | 81.3 | 81.3 | | MathVista | 87.0 | 87.8 | | OCRBench | 87.6 | 86.5 | | MathVision | 83.3 | 81.3 | | MMBench | 91.2 | 91.0 |
|
Model usage
The following package versions are used:
transformers>=5.12.1- a recent
torch build (matching your CUDA)
vllm>=0.24.0 (for serving)- optional:
flash-linear-attention and causal-conv1d for faster linear-attention inference
The model reasons by default, emitting a chain-of-thought inside <think> ... </think>.
Transformers
1from transformers import AutoProcessor, AutoModelForMultimodalLM
2
3model_id = "issai/Qwen3.5-35B-A3B-Kazakh"
4processor = AutoProcessor.from_pretrained(model_id)
5model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype="auto", device_map="auto")
6
7messages = [
8 {"role": "user", "content": [{"type": "text", "text": "Балқаш көлінің қандай ерекшелігі бар?"}]}
9]
10
11inputs = processor.apply_chat_template(
12 messages, add_generation_prompt=True, tokenize=True,
13 return_dict=True, return_tensors="pt",
14).to(model.device)
15
16outputs = model.generate(
17 **inputs, max_new_tokens=32768,
18 do_sample=True, temperature=1.0, top_p=0.95, top_k=20,
19)
20
21print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
Image input:
1messages = [
2 {"role": "user", "content": [
3 {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
4 {"type": "text", "text": "Суретте не бейнеленген?"},
5 ]}
6]
7
8inputs = processor.apply_chat_template(
9 messages, add_generation_prompt=True, tokenize=True,
10 return_dict=True, return_tensors="pt",
11).to(model.device)
12
13outputs = model.generate(
14 **inputs, max_new_tokens=32768,
15 do_sample=True, temperature=1.0, top_p=0.95, top_k=20,
16)
17
18print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
To disable thinking, pass enable_thinking=False to apply_chat_template.
vLLM
Serve an OpenAI-compatible endpoint:
1vllm serve issai/Qwen3.5-35B-A3B-Kazakh \
2 --served-model-name qwen \
3 --reasoning-parser qwen3 \
4 --max-model-len 90000 \
5 --enable-expert-parallel \
6 --max-num-seqs 64
1from openai import OpenAI
2
3client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
4
5resp = client.chat.completions.create(
6 model="qwen",
7 messages=[{"role": "user", "content": """Төмендегі экзопланеталардың қайсысында тығыздығы ең жоғары?
8a) Жермен бірдей құрамы бар, бірақ Жерден 5 есе ауыр планета.
9b) Жермен бірдей құрамды, бірақ Жерден екі есе жеңіл планета.
10c) Массасы Жерден 2 есе үлкен және тығыздығы шамамен 5.5 г/см^3 болатын планета.
11d) Жердің массасы мен радиусымен тең планета."""}],
12 max_tokens=81920, temperature=1.0, top_p=0.95, presence_penalty=1.5,
13 extra_body={"top_k": 20},
14)
15
16print(resp.choices[0].message.reasoning) # chain-of-thought
17print(resp.choices[0].message.content) # final answer
Image input:
1import base64
2
3image = base64.b64encode(open("image.png", "rb").read()).decode()
4
5resp = client.chat.completions.create(
6 model="qwen",
7 messages=[{"role": "user", "content": [
8 {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}},
9 {"type": "text", "text": "Суретте қандай фигуралар бар?"},
10 ]}],
11 max_tokens=81920, temperature=1.0, top_p=0.95, presence_penalty=1.5,
12 extra_body={"top_k": 20},
13)
14
15print(resp.choices[0].message.content)
To disable thinking, add "chat_template_kwargs": {"enable_thinking": false} to extra_body.
License
This model is released under the
Apache 2.0 license, inherited from the original
Qwen3.5 model.
