The q4f16_0 layout (KN) is dramatically faster on Qualcomm Adreno GPUs compared to q4f16_1 (NK). The q4f16_1 format requires weight transposition operations that are catastrophically slow on Adreno's OpenCL implementation, resulting in 6-10x slower inference.

Source

Original model: Qwen/Qwen2.5-Coder-1.5B-Instruct (Apache-2.0)
Quantization source: mlc-ai/Qwen2.5-Coder-1.5B-Instruct-q4f16_1-MLC
Requantized by: NavixMind using mlc_llm package with mobile overrides (context_window_size=4096, prefill_chunk_size=1024, max_batch_size=1)

Only the weight layout was changed (q4f16_1 → q4f16_0). The model architecture, tokenizer, and training are identical to the original.

Usage with MLC LLM

Download the model weights and place them in your MLC models directory. Use model_lib: qwen2_q4f16_0_1be22ffdc6429c5019af9af8dae22086 when loading.

License

Apache-2.0 (same as the original Qwen2.5-Coder-1.5B-Instruct model).

alexandertaboriskiy/Qwen2.5-Coder-1.5B-Instruct-q4f16_0-MLC

Qwen2.5-Coder-1.5B-Instruct — q4f16_0 (MLC LLM)

What is this?

Why q4f16_0 instead of q4f16_1?

Source

Usage with MLC LLM

License

alexandertaboriskiy/
Qwen2.5-Coder-1.5B-Instruct-q4f16_0-MLC