FP8_DYNAMIC (W8A8) quantization of joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive (architecture Qwen3_5MoeForConditionalGeneration, base model Qwen/Qwen3.6-35B-A3B) — a 35B-A3B MoE roleplay finetune.

Source: joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive
License: apache-2.0 (inherited from the source and base models)
Tool: llm-compressor, data-free one-shot
Scheme: FP8 per-channel weights (static), FP8 per-token activations (dynamic)
Weights size: 36.66 GiB (down from ~70 GiB BF16) — fits a single 96 GB GPU with ample room for KV cache

Kept in BF16

The following modules were excluded from quantization: lm_head, embeddings, the vision tower (model.visual.*), the Gated DeltaNet linear-attention layers (linear_attn.*), MoE routers (mlp.gate), the shared-expert gate (shared_expert_gate) and the MTP head (mtp.*). The ignore list matches the reference RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic quantization.

Recipe

python

from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "joyfox/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive"
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=[
        "re:.*lm_head",
        "re:visual.*",
        "re:model.visual.*",
        "re:.*mlp.gate$",
        "re:.*embed_tokens$",
        "re:.*shared_expert_gate$",
        "re:.*linear_attn.*",
        "re:^mtp.*",
    ],
)
oneshot(model=model, recipe=recipe)
model.save_pretrained("out")
processor.save_pretrained("out")

Serving with vLLM

bash

vllm serve cloud19/JoyFox-Qwen3.6-35B-A3B-RP-Aggressive-FP8-Dynamic \
  --served-model-name joyfox \
  --enable-chunked-prefill \
  --max-model-len 16384 --kv-cache-dtype fp8 \
  --limit-mm-per-prompt '{"image": 0, "audio": 0, "video": 0}' \
  --default-chat-template-kwargs '{"enable_thinking": false}'

Notes:

Chunked prefill is required. The hybrid attention stack (Gated DeltaNet linear-attention layers) uses vLLM's mamba cache in mode align; passing --no-enable-chunked-prefill makes the engine refuse to start.
The MTP head is shipped as model_mtp.safetensors, but speculative decoding is intentionally not used here.
Thinking (<think>) is disabled at the chat-template level via enable_thinking: false.