A vision-language model created by grafting the vision tower of
/unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit onto the language model of mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit.
The finetuned model was language-only (its vision tower had been stripped
during training), but was the best model for agentic coding i ever tested for 32gb systems (very
good multi turn tool calling and small thinking blocks for a qwen variant).
This merge fully restores multimodal capability by re-attaching the original vision encoder and
projector from the base VLM, while keeping the finetuned language weights intact.
python -m mlx_vlm.generate \
--model YOUR_MODEL_NAME \
--image path/to/image.jpg \
--prompt "Describe this image. Extract all text" \
--max-tokens 5000
Python:
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
model, processor = load("YOUR_MODEL_NAME")
config = model.config
prompt = apply_chat_template(
processor, config,
"Describe this image. Extract all text",
num_images=1,
)
output = generate(model, processor, prompt, ["path/to/image.jpg"],
max_tokens=5000, verbose=True)
print(output)
How the merge was done
The two source models share the same qwen3_5_moe architecture and an
identical namespace convention, which made a clean weight-level merge
possible without any retraining, fine-tuning, or projector re-alignment.
1. Namespace split
All tensors in both models fall into two disjoint prefixes:
language_model.* -> Language model + LM head -> taken from the FINETUNE
vision_tower.* -> Vision encoder + merger -> taken from the BASE VLM
The merge is a straightforward union: every language_model.* tensor from the
finetune, and every vision_tower.* tensor from the base VLM.
2. Dimension compatibility
The vision projector (vision_tower.merger) outputs
vision_config.out_hidden_size = 2048, which matches the language model's
text_config.hidden_size = 2048. Vision features therefore project directly
into the LM embedding space with no adapter needed. Image features are
injected at the image_token_id position (single-point injection;
deepstack_visual_indexes is empty).
3. Precision and quantization
Language model: retains the finetune's mixed quantization scheme --
4-bit switch_mlp experts, 8-bit attention / shared experts / embeddings /
LM head (group size 64, affine). The finetune's quantization map was
carried over verbatim.
Vision tower: kept at full precision (bfloat16), exactly as shipped in the
base VLM. MLX supports this mixed quantized-LM / bf16-vision setup.
4. Configuration
The merged config.json uses the finetune's config as the base (it holds the
correct quantization map and text_config), with the following vision-related
fields grafted in from the base VLM:
vision_config
image_token_id, video_token_id
vision_start_token_id, vision_end_token_id
The multi-token-prediction head was not carried over
(mtp_num_hidden_layers remains 0, matching the finetune). The image preprocessor
(preprocessor_config.json) were taken from the base VLM, since the finetune's tokenizer
configuration was text-only. The tokenizer vocabulary is identical between both sources
(vocab_size = 248320, same special-token IDs), so the base VLM's template is
fully compatible.
5. Verification
The merged weights were verified byte-for-byte via SHA-256 hashes of each
tensor, comparing the merged output against both sources:
All 1757 language_model.* tensors are byte-identical to the finetune.
All 333 vision_tower.* tensors are byte-identical to the base VLM.
This model is a derivative combining:
Language model: mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit
Vision tower: unsloth/Qwen3.6-35B-A3B-UD-MLX-4bit
License: apache-2.0. You must comply with the licenses of BOTH source
models. Please cite and credit the original authors of both the base VLM and
the finetune.
Reproduction
The merge was performed with a script (scripts/merge.py) that:
Loads all safetensors from both source directories natively in MLX
(preserving bfloat16 and packed-quant dtypes).
Selects language_model.* tensors from the finetune and vision_tower.*
tensors from the base VLM.
Builds the merged config (finetune config + base vision fields).
Copies the image preprocessor config from the base VLM, rest from finetune.
Saves the merged safetensors with a model.safetensors.index.json.
The merge was split into Hugging Face splittensors using the scripts/split.py.
The model was verified for correct tensors (checking text tensors were untouched) using the scripts/verify.py.
You should be able to do your own merges with the script as long as the two models are compatible (check for
same dimensions, same tokenizer size). Verify and check vision capabilities (f.e. ocr a text).
The model uses unsloth chat_template with default reasoning_preserve=true.
If you prefer the original chat_template replace chat_template.json with chat_templat.json.org or any you like.
Recommended settings (coding/thinking):
Though finetuned, this is still Qwen3.6, so stick close to the official Qwen recommendations
(temp 0.7, top_p=0.95, min_p=0.0, top_k=20, repetition_penalty=1.0, presence_penalty=0.0).
Avoid temp < 0.4, as this can cause the model to loop!!!
My personal recommendation (forced! in oMLX when required, as f.e. copilot likes to use temp 0.0):
temp 0.6, top_p=0.85, min_p=0.0, top_k=20, repetition_penalty=1.0, presence_penalty=0.0,
thinking_budget=none, enable_thinking=true, preserve_thinking=true, output_token_limit=16384
