Component	Scheme	Details
Attention Layers	FP8_DYNAMIC	W8A8, preserves precision for Q/K/V/O projections
MLP Layers	NVFP4	W4A4, optimizes latency for gate/up/down projections
Vision Encoder	BF16 (unquantized)	Full precision for visual understanding
LM Head	BF16 (unquantized)	Full precision for output quality

SmoothQuant

Applied with strength 0.8 for activation smoothing before quantization:

Q/K/V projections ← input_layernorm
Gate/Up projections ← post_attention_layernorm

Model Details

Base Model: Qwen/Qwen3-VL-8B-Instruct
Quantization Method: compressed-tensors (llm-compressor)
Model Size: ~7.7 GB (reduced from ~17 GB, ~55% compression)
Config Groups: FP8_DYNAMIC (attention) + NVFP4 (MLP)

Usage with vLLM

bash

vllm serve JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4-FP8-Dynamic \
    --quantization compressed-tensors \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.9 \
    --enforce-eager \
    --max-model-len 8192

python

from vllm import LLM, SamplingParams

llm = LLM(
    model="JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4-FP8-Dynamic",
    quantization="compressed-tensors",
    trust_remote_code=True,
    kv_cache_dtype="fp8",
    max_model_len=8192,
)

sampling_params = SamplingParams(temperature=0.7, top_p=0.95)
prompts = ["Your prompt here"]
outputs = llm.generate(prompts, sampling_params)

Usage with Transformers

python

from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
import torch

model_id = "JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4-FP8-Dynamic"
processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

Quantization Config

Target Patterns (vLLM fused-layer compatible)

Attention (FP8_DYNAMIC): self_attn.(q_proj|k_proj|v_proj|o_proj|qkv_proj|qkv)
MLP (NVFP4): mlp.(gate_proj|up_proj|down_proj|gate_up_proj)
Ignored: Vision encoder (visual.*), output head (lm_head), MoE gates (mlp.gate)

Recipe Structure (`recipe.yaml`)

SmoothQuantModifier: Activation smoothing (strength=0.8)
QuantizationModifier (attn): FP8_DYNAMIC for attention layers
QuantizationModifier (mlp): NVFP4 for MLP layers

Performance Notes

Model loads in ~4 seconds, uses ~8 GiB VRAM
On 24GB GPU: ~10 GiB available for KV cache (~159K tokens)
GPU without native FP4/FP8 support falls back to Marlin kernels
Requires vLLM 0.11+ with compressed-tensors support

License

Apache 2.0, same as the base model.

JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4-FP8-Dynamic

Qwen3-VL-8B-Instruct-NVFP4-FP8-Dynamic

Quantization Strategy

SmoothQuant

Model Details

Usage with vLLM

Usage with Transformers

Quantization Config

Target Patterns (vLLM fused-layer compatible)

Recipe Structure (recipe.yaml)

Performance Notes

License

JEILDLWLRMA/
Qwen3-VL-8B-Instruct-NVFP4-FP8-Dynamic

Recipe Structure (`recipe.yaml`)