Model Card for Qwen3.5-4B (4-bit GGUF, with vision)
A 4-bit GGUF build of Qwen3.5-4B packaged to run entirely on device, with the
vision tower, the tokenizer, the merge table, the chat template and the
sampling card all embedded in the single weight file. Nothing calls out to a
server at inference time and there are no sibling files to keep in sync.
This repository holds the quantized weights only, not training data or the
original checkpoint.
File format
.ggxf is GGUF plus the tensors and metadata keys this project adds, so these
files load in Gadeon and are not expected to load in other GGUF readers.
Files
| File | Purpose |
|---|
Qwen3.5-4B-UD-Q4_K_XL.ggxf | everything: weights, vision tower, tokenizer, chat template, sampling card |
One file, nothing beside it. The vision tower travels inside the GGUF rather
than as a separate mmproj, and the sampling parameters travel with it, so a
runtime never has to guess them or borrow another model's settings.
Every block type here is a standard ggml one, inherited from the upstream
quantization. The additions are the embedded vision tower and the metadata,
and those have only been exercised in the companion on-device runtime --
third-party tools have not been tested against this file, and a tool that
does not expect a tower inside the weight file may ignore it or refuse it.
Provenance
The language trunk and the multi-token-prediction drafter are
unsloth's
UD-Q4_K_XL quantization with the MTP
block of Qwen3.5-4B,
used as published. What this build adds is the vision tower and the metadata: the
v.* /
mm.* tensors and
clip.* configuration, and the generation config
with its sampling matrix and its stop-token list. The tower is carried at
source precision (F16/F32).
Model Details
Model Description
Qwen3.5-4B is a dense hybrid. Its layers alternate three Gated DeltaNet
linear-attention blocks to one full attention block. The linear blocks carry a
small fixed-size recurrent state instead of a growing key/value cache, so
memory stays flat as context grows, while the periodic attention blocks
preserve exact long-range recall.
This build is multimodal. The vision tower is included, so image input
works from this file alone.
This build carries the multi-token-prediction layer (blk.32, the
nextn block), so a runtime that speculates can draft from it; one that
does not decodes normally.
- Developed by: Qwen, Alibaba (base model); quantization by unsloth;
packaging by leok7v
- Model type: Hybrid Gated DeltaNet + attention causal language model with
a vision encoder
- Language(s): English and the languages of the base model
- License: Apache 2.0, inherited from the base model
- Quantized from model: Qwen/Qwen3.5-4B
Model Sources
Uses
Direct Use
On-device chat, summarization, drafting, question answering over supplied text,
and description of supplied images, in offline or privacy-sensitive settings
where prompts must not leave the device.
Out-of-Scope Use
Anything the base model's licence and card exclude. This build adds no
alignment, no safety tuning and no filtering of its own; it is the upstream
model in a different numeric format. Not a basis for legal, medical, financial
or safety decisions, and not intended for high-stakes automated decisions
without human review.
Bias, Risks, and Limitations
The base model's biases and failure modes carry over unchanged.
Four bits is a mild compression compared with the 1-bit and 2-bit builds in
this family, but it is still lossy: outputs will not match the upstream model
token for token, and the gap is widest where the model was already uncertain.
Evaluate on your own task rather than assuming it behaves like the
full-precision model.
Recommendations
Treat generations as drafts to verify, not as fact. For anything where
accuracy matters, and for arithmetic in particular, check the result.
How to Get Started with the Model
The tokenizer, the chat template and the sampling card are embedded, so a
runtime reads all of them from the GGUF. The chat template documents the
roles, the thinking block and the tool-call format.
Generation stops on either <|im_end|> (248046) or <|endoftext|> (248044).
Both ids are carried in the embedded generation config and in
tokenizer.ggml.eos_token_ids; a runtime that reads only the scalar
tokenizer.ggml.eos_token_id will see <|im_end|> alone.
Technical Specifications
Model Architecture and Objective
| Property | Value |
|---|
| Objective | Autoregressive next-token prediction |
| Hidden size | 2560 |
| Layers | 32 (Gated DeltaNet + attention, three to one) |
| Attention heads | 16 query / 4 key-value |
| Head dimension | 256 (partial rotary, 64) |
| Linear-attention state | 128, 16 groups, inner size 4096 |
| Feed-forward | 9216 |
| Vocabulary | 248320 |
| Context length | up to 262144 tokens |
| Vision encoder | 24 blocks, 768px tiles, 2x2 spatial merge |
Weight format
3.41 GiB, 739 tensors (298 of them the vision tower at
F16/F32), GGUF v3. The trunk is a MIXED-precision assignment rather than one
block type throughout: the widths are chosen per tensor, so the most sensitive
rows keep more bits than the bulk of the model.
Citation
Please cite the upstream Qwen3.5-4B model. See the base model card at
Qwen/Qwen3.5-4B for the canonical reference
and license.
Model Card Contact
leok7v on Hugging Face.
