You can deploy this model and start earning money today!

daydreamwarrior/
QwenPaw-Flash-9B-heretic-coder3101-GGUF

Name: QwenPaw-Flash-9B-heretic-coder3101-GGUF
Availability: InStock

gguf

agentscope-ai/QwenPaw-Flash-9B

quantized

endpoints_compatible

conversational

Views

No views yet

The QwenPaw-Flash-9B-heretic-mmproj-Q8_0.gguf model provides a full Q8_0 quantization. In contrast, the original version falls back to FP16 for the ffn_down weights because its dimension is not divisible by 32. This version resolves the issue by padding the dimension from 4,304 to 4,320.

Q4_K_L is based on observations of unsloth Q4_K_XL and unsloth Q4_K_M, without imatrix.

llama-quantize \
    --tensor-type ssm_alpha=F32 \
    --tensor-type ssm_beta=F32 \
    --tensor-type ssm_out=q8_0 \
    --tensor-type attn_gate=q5_k \
    --tensor-type attn_qkv=q5_k \
    --tensor-type attn_v=q6_k \
    ./QwenPaw-Flash-9B-heretic-BF16.gguf ./QwenPaw-Flash-9B-heretic-Q4_K_L.gguf Q4_K_M

Q6_K_S is based on observations of unsloth Q5_K_XL, without imatrix. recommended.

llama-quantize \
    --tensor-type ssm_alpha=F32 \
    --tensor-type ssm_beta=F32 \
    --tensor-type ssm_out=q8_0 \
    --tensor-type ffn_gate=q5_k \
    --tensor-type ffn_up=q5_k \
    ./QwenPaw-Flash-9B-heretic-BF16.gguf ./QwenPaw-Flash-9B-heretic-Q6_K_S.gguf Q6_K

Q6_K_L is based on observations of unsloth Q6_K, without imatrix.

llama-quantize \
    --tensor-type ssm_alpha=F32 \
    --tensor-type ssm_beta=F32 \
    --tensor-type ssm_out=q8_0 \
    ./QwenPaw-Flash-9B-heretic-BF16.gguf ./QwenPaw-Flash-9B-heretic-Q6_K_L.gguf Q6_K

daydreamwarrior/QwenPaw-Flash-9B-heretic-coder3101-GGUF

daydreamwarrior/
QwenPaw-Flash-9B-heretic-coder3101-GGUF