You can deploy this model and start earning money today!

aaaljaz/
qwen3.8-27b-ninfer-minq4

Name: qwen3.8-27b-ninfer-minq4
Availability: InStock

Views

No views yet

Qwen3.8-27B — NInfer min-Q4 artifact (A5000/sm_86 fork)

⚠️ Requires the NInfer sm_86 fork to load. This is a .ninfer container, NOT a GGUF — llama.cpp, vLLM, LM Studio etc. cannot read it. It was built for (and measured on) a 16 GB RTX A5000 Laptop GPU. The reference fork: a port of Neroued/ninfer to sm_86 with Q4-only groupwise kernels, pinned-host embedding, and INT4 KV support.

What it is

A custom-format conversion of Qwen/Qwen3.8-27B built with the fork's converter:

All text weights Q4G64_F16S (4-bit symmetric, fp16 group-64 scales), except:
token_embedding: Q6G64_F16S — at load time the engine places this in pinned host memory (zero-copy gather; the embedding is read one ~10 KB row per token, so PCIe cost is nil) — device-resident weights ≈ 12.71 GiB
Norms/control tensors BF16/FP32; vision tower and MTP package host-resident unless --vision / --spec mtp are requested

Measured performance (RTX A5000 Laptop 16 GB, sm_86)

Config	Context	Decode	Prefill
plain, int4 KV (`--kv-dtype i4`)	131,072	12.6 tok/s	~380 tok/s
MTP3, int8 KV	49,152	25.7 tok/s	~380 tok/s
MTP3, int4-g128 KV	122,880	22.4 tok/s	~380 tok/s

Quality

Wikitext-2 (50×512 chunks, llama.cpp perplexity harness on equivalent GGUF simulation of the exact same quantization arithmetic):

Model	PPL
Qwen3.8-27B BF16 (fp32 ref)	6.012
unsloth IQ3_XXS GGUF (anchor)	6.256
this artifact (min-Q4)	6.299

All 4-bit KV variants ≈ +0.003 PPL vs int8 KV (measured, negligible).

Usage (fork CLI)

bash

ninfer qwen3_8_27b_minq4.ninfer \
  --kv-dtype i4 --max-context 122880 --kv-capacity 122880 \
  --prefill-chunk 64 --spec mtp --draft-tokens 3 \
  --prompt "..."

Format details

Built from the BF16 source with tools/convert/qwen3_8_27b/convert.py (fork's _a5000_endpoint_and_down inventory: Q4 everywhere except Q6 embedding + Q4 head). Container layout matches the official qwen3.8-27b/groupwise-int identity, so the official 5090 engine will also load it — but the all-Q4 layout is tuned for 16 GB cards and is untested on sm_120a.

Source / engine

The engine that loads this artifact is the sm_86 fork of NInfer: aljazceru/ninfer — every change vs upstream is marked // Fork:. The official Neroued/ninfer targets RTX 5090 (sm_120a) and will also load this container but is untested with this layout.

License

Apache 2.0, inherited from the base model. Quantization performed with NInfer's converter (AGPL engine; the artifact itself is a conversion of Apache-2.0 weights).

aaaljaz/qwen3.8-27b-ninfer-minq4