docker run -d --name taide-fp8 --restart unless-stopped \
  --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HF_TOKEN=$HF_TOKEN \
  -e VLLM_USE_DEEP_GEMM=0 \
  vllm/vllm-openai:latest \
  --model RickYuan/Gemma-3-TAIDE-12b-Chat-2602-FP8-Dynamic \
  --served-model-name taide-fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.5 \
  --language-model-only

On DGX Spark / GB10 (sm_121), use the aarch64 image and pin the attention backend:

docker run -d --name taide-fp8 --restart unless-stopped \
  --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_USE_DEEP_GEMM=0 \
  vllm/vllm-openai:nightly-aarch64 \
  --model RickYuan/Gemma-3-TAIDE-12b-Chat-2602-FP8-Dynamic \
  --served-model-name taide-fp8 \
  --max-model-len 32768 \
  --attention-backend triton_attn \
  --gpu-memory-utilization 0.28 \
  --language-model-only

Native

pip install vllm
VLLM_USE_DEEP_GEMM=0 vllm serve RickYuan/Gemma-3-TAIDE-12b-Chat-2602-FP8-Dynamic \
  --served-model-name taide-fp8 \
  --max-model-len 32768 \
  --language-model-only

Flags explained

Flag	Why
`--language-model-only`	Skips multimodal init. The base architecture is `Gemma3ForConditionalGeneration`, but TAIDE was not trained on image data. Without this, vLLM looks for a `preprocessor_config.json` that this repo does not ship.
`--gpu-memory-utilization`	Fraction of GPU memory reserved for weights + KV cache. Weights are ~14 GB; the rest becomes KV cache. On a 24 GB card use `0.9`; on DGX Spark's 128 GB unified pool, `0.28` (~34 GB) is plenty and leaves room for the OS.
`VLLM_USE_DEEP_GEMM=0`	DeepGEMM's FP8 kernels are unstable on consumer Blackwell (sm_120/121). Harmless elsewhere.
`--attention-backend triton_attn`	flash_attn crashes on sm_121. Omit on other hardware.

Do not pass --quantization fp8. The compressed-tensors format is auto-detected. Adding the flag can route the model through the online quantization path instead of loading the pre-quantized weights.

Querying the server

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "taide-fp8",
    "messages": [{"role": "user", "content": "請用三句話介紹阿里山。"}],
    "max_tokens": 512
  }'

OpenAI-compatible clients work as-is:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="taide-fp8",
    messages=[{"role": "user", "content": "請用三句話介紹阿里山。"}],
)
print(resp.choices[0].message.content)

Offline inference

from vllm import LLM, SamplingParams

llm = LLM(
    model="RickYuan/Gemma-3-TAIDE-12b-Chat-2602-FP8-Dynamic",
    max_model_len=32768,
)
out = llm.chat(
    [{"role": "user", "content": "請用三句話介紹阿里山。"}],
    SamplingParams(max_tokens=512),
)
print(out[0].outputs[0].text)

Memory footprint

Component	Size
Weights (FP8)	~14 GB
CUDA graphs + activations	~3 GB
KV cache	remainder of the budget

At --gpu-memory-utilization 0.28 on a 128 GB DGX Spark, this yields ~17.6 GB of KV cache (~161K tokens, ~4.9 concurrent 32K-context requests). Raise the value for more concurrency, or add --kv-cache-dtype fp8 to roughly double KV capacity.

Quantization environment

Component	Version
llm-compressor	0.13.0
compressed-tensors	0.18.0
transformers	5.15.0
torch	2.13.0+cu130
vLLM (validation)	0.26.1
Hardware	NVIDIA DGX Spark (GB10, sm_121, aarch64)

Older llm-compressor releases (0.12.x) fail against transformers 5.x with an ImportError on GraniteMoeParallelExperts; 0.13.0 requires compressed-tensors >= 0.18. Installing without --no-deps on a vLLM base image resolves this correctly while leaving the sm_121-capable torch build in place.

Quantization recipe

from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model_id = "taide/Gemma-3-TAIDE-12b-Chat-2602"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

oneshot(model=model, recipe=QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=["lm_head", "re:.*vision_tower.*", "re:.*multi_modal_projector.*"],
))

Performance

Roughly 16 tok/s single-request decode on a DGX Spark (vLLM 0.26.1, 32K context, --gpu-memory-utilization 0.28, ~161K token KV cache). The Spark's unified memory supports higher concurrency, so aggregate throughput under parallel load is considerably better than the single-stream figure.

No formal benchmark comparison against the BF16 original has been run. Spot checks on Taiwanese colloquialisms and general knowledge produced fluent Traditional Chinese output with no observable degradation, but users should evaluate against their own workload.

License

This is a derivative of a TAIDE model. Use is subject to both the TAIDE model license and the Google Gemma Terms of Use.

Quantization introduced no additional training data and does not alter the model's intended behavior, though some quality variation is inherent to the process. All limitations and disclaimers of the base model apply.