Views
No views yet
FP8_DYNAMIC weight-only FP8 quantization via
llmcompressor. Saved in compressed-tensors format with FP8
weights (*.safetensors) and quantization_config in config.json,
so sgl.Engine loads it directly with no bf16 staging transient.FP8_DYNAMIC quantization on every nn.Linear weight in the
backbone (lm_head is NOT quantized — output projection numerics
matter).nla_meta.yaml, prompt templates, model card,
license, generation config, tokenizer extras, …) are preserved
verbatim. The upstream README.md is renamed to README_upstream.md.apps/nla/server.py1export NLA_VERBALIZER_MODEL=<this repo>
2export NLA_VERBALIZER_QUANTIZATION=compressed-tensors
3# Do NOT set NLA_FP8_VERBALIZER — it is overridden by the explicit
4# --verbalizer-quantization (or NLA_VERBALIZER_QUANTIZATION) anyway.
5uv run server.py --truncate-source --fp8-source --int4-reconstructorapps/nla/server.py will pass quantization="compressed-tensors" to
sgl.Engine, which loads the pre-quantized FP8 weights with no bf16
load-time peak. See the apps/nla README's "Multi-GPU > Layout E" for
the full per-GPU memory budget.apps/nla/build_quantized_models.py --target verbalizer-fp8AffineQuantizedTensor-backed) FP8.
Sglang 0.5.x has no "torchao" entry in its quantization registry, so
that output is not loadable by apps/nla/server.py. Use this
checkpoint for the sglang verbalizer; reserve the torchao path for the
HF source/reconstructor (which transformers consumes natively).