This model is a small version of Qwen/Qwen3.5-4B. It operates with MLX on Apple
silicon. The model reads text and images. The model has a reasoning mode.
Most of the weights use 4 bits. Some weights use 6 bits or 8 bits. No weight uses
3 bits.
The size of the model is 2660 MB.
2. Why this model is different
Qwen3.5 is a hybrid model. It has two types of layer:
8 layers use full attention.
24 layers use a gated DeltaNet.
The usual MLX recipe is mixed_3_6. This recipe is for older models. It gives more
bits to v_proj, to down_proj and to lm_head. The DeltaNet layers do not have
these three parts. Thus the recipe gives 3 bits to all the weights in the 24
DeltaNet layers.
Two of these weights are in_proj_a and in_proj_b. They control the memory of
the model. The model calculates the memory decay with this equation:
decay = exp(-exp(A_log) * softplus(a + dt_bias))
The equation contains two exponential functions. Thus a small error in a becomes
a large error in the decay. The decay multiplies the memory at each token. Thus the
error increases when the text becomes longer.
The two weights are only 3.9 M parameters. This is 0.06 % of the model. But they
have a large effect on the quality.
3. The changes in this model
Part of the model
Bits
Group size
in_proj_a, in_proj_b (memory control)
8
32
down_proj (on the standard schedule), v_proj
6
64
All the other language weights
4
64
Vision tower
4
64
Embedding (tied to the output)
4
64
The usual tools do not compress the vision tower. In this model the vision tower
uses 4 bits. This change makes the vision tower smaller by 473 MB (from 667 MB to
194 MB). The model uses these bytes for the language layers.
4. Test data
The tests use the bf16 model as the reference. The text tests use the wikitext-2
test set. The tests use 262144 tokens and a sequence length of 2048.
Measurement
mixed_3_6
mlx-community 4-bit
This model
Peak memory (full model, with images)
3.730 GB
3.935 GB
3.188 GB
Memory after the load
2.694 GB
2.827 GB
2.478 GB
Size on disk
2892 MB
3034 MB
2660 MB
Perplexity (bf16 = 10.139)
13.982
10.515
10.363
Increase of the perplexity
+37.9 %
+3.7 %
+2.2 %
KL divergence from bf16
0.334
0.102
0.0935
Agreement with bf16 (first token)
75.6 %
86.1 %
86.7 %
Tokens before a different output
3.16
5.21
5.62
Decode speed
60.4 tok/s
166.1 tok/s
160.0 tok/s
This model is better than the 4-bit model in the quality tests. It is also 374 MB
smaller.
The mixed_3_6 model is slow. MLX does not have fast 3-bit code. This model is
2.65 times faster than the mixed_3_6 model.
Long text
The tests measure the KL divergence at each position in a sequence of 8192 tokens.
The error increases from the start of the sequence to the end.
Model
Start
End
Increase
mixed_3_6
0.311
0.437
+0.126
This model
0.088
0.152
+0.064
5. How to use the model
Install mlx-vlm. Then obey these steps.
Download the model.
Give the path of the model to the load function.
Give a prompt and an image to the generate function.
python
1from mlx_vlm import load, generate
2from mlx_vlm.prompt_utils import apply_chat_template
34model, processor = load("Spakie/Qwen3.5-4B-MLX-4bit-hybrid")5prompt = apply_chat_template(processor, model.config,"What is in this image?", num_images=1)6print(generate(model, processor, prompt,["image.png"], max_tokens=200))
For text only, use mlx-lm.
python
1from mlx_lm import load, generate
23model, tokenizer = load("Spakie/Qwen3.5-4B-MLX-4bit-hybrid")4print(generate(model, tokenizer,"Tell me about Apple silicon.", max_tokens=200))
6. Limits
The tests for the perplexity and the KL divergence use text only. The tools for
these tests do not load the vision tower.
The tests of the images use three images. These tests do not give a score.
The tests do not include a standard benchmark set, for example MMLU.
The model keeps the context length of the initial model.
