Base model: CrucibleLab/L3.3-70B-Loki-V2.0
Quantization: NVFP4 with FP8 KV cache
Generated with: nvidia-modelopt on B200 hardware
Tested with: vLLM on GB10 DGX Spark (SM121)
Launch command (on SM121):
vllm serve ~/models/L3.3-70B-Loki-V2.0-nvfp4
--max-model-len 65536
--host 0.0.0.0
--port 8000
--gpu-memory-utilization 0.50
--kv-cache-dtype fp8
Quantization Notes
Weights and calibration scales were generated using nvidia-modelopt (hf_ptq.py) with NVFP4 precision and FP8 KV cache quantization.
Calibration was performed on B200 hardware. KV cache scales (k_scale/v_scale) were subsequently extracted and injected into the checkpoint
on a GB10 DGX Spark (SM121) to produce a self-contained HuggingFace-compatible checkpoint.
Known issues
TRT-LLM PyTorch backend produces incoherent output on SM121
The checkpoint does not include a calibrated q_scale for the FP8 KV cache. vLLM will warn at startup that q_scale is being set to k_scale
as a fallback. This is expected and has not been observed to cause meaningful quality degradation in practice.
HuggingFace's model size display shows ~36B parameters due to NVFP4 weight packing. The actual model is 70B parameters (CrucibleLab/L3.3-70B-Loki-V2.0).
