Token-efficient, quantized reasoning model based on Qwen3.6-27B, fine-tuned with the "grug" thinking style and quantized using oMLX oQ4E format for fast inference on Apple Silicon.
grug-27b-oQ4E-fp16 is a quantized variant of
ProCreations/grug-27b, itself derived from
Qwen/Qwen3.6-27B. The model has been:
While the original grug-27b model uses bfloat16, this variant converts to float16 (fp16) as the quantized base dtype. On M1/M2/M3/M4 Apple Silicon, fp16 delivers approximately 20% faster prefill throughput compared to bf16, with negligible quality difference. The quantized weights remain at ~4.8 BPW via oQ4E.
1from mlx_lm import load, generate
2
3model, tokenizer = load("your-username/grug-27b-oQ4E-fp16")
4response = generate(
5 model, tokenizer,
6 prompt="Your prompt here",
7 max_tokens=1024,
8 temp=0.6,
9 top_p=0.95
10)
11print(response)
1mlx_lm.run \
2 --model your-username/grug-27b-oQ4E-fp16 \
3 --max-tokens 4096 \
4 --temp 0.6 \
5 --top-p 0.95
This model card and quantization are released under the
Apache License 2.0. The base model
Qwen/Qwen3.6-27B and intermediate
ProCreations/grug-27b are also licensed under Apache 2.0.
1@misc{grug27b,
2 title = {grug-27b-oQ4E-fp16},
3 author = {Your Username},
4 year = {2025},
5 howpublished = {\url{https://huggingface.co/your-username/grug-27b-oQ4E-fp16}},
6 note = {Quantized from ProCreations/grug-27b (Apache 2.0), based on Qwen/Qwen3.6-27B}
7}
