Views
No views yet
shift(1)
fix; clean O(n) implementations with tests.1vllm serve <this-repo> --tensor-parallel-size 2 --max-model-len 32768 \
2 --gpu-memory-utilization 0.92
3# NVFP4 is auto-detected; no --quantization flag needed.
4# On no-P2P multi-GPU boxes add: NCCL_P2P_DISABLE=1, --disable-custom-all-reduceqwen3_5_moe quantizes cleanly with llm-compressor
(targets=Linear, scheme NVFP4) with these ignores:
lm_head, re:.*conv1d.* (DeltaNet conv), re:.*mlp.gate$ and
re:.*shared_expert_gate$ (MoE routers), re:.*mtp.*.GraniteMoeParallelExperts which llm-compressor
still imports — a dummy class injected before import satisfies it safely for
non-Granite models.