This is a modified, quantized derivative. The weights have been AWQ 4-bit quantized; everything else (architecture, tokenizer, chat template) is unchanged from the base model. It loads on the TensorRT-LLM PyTorch (_torch) backend, which reads hf_quant_config.json and dispatches the W4A16-AWQ kernels — no engine build step.

Quantization

Method: W4A16-AWQ (4-bit weights, 16-bit activations), group size 128, with per-input pre-quant scales. lm_head kept in higher precision.
Tooling: NVIDIA TensorRT Model Optimizer (modelopt 0.37) INT4_AWQ_CFG, AWQ scale search calibrated on cnn_dailymail, exported via export_hf_checkpoint.
Format: modelopt unified-HF checkpoint (quant_algo: W4A16_AWQ).

Hardware

Ada (RTX 40-series, sm_89)+: autotuner on. Ampere (RTX 30-series, sm_86): autotuner disabled automatically (BareMetalRT handles this per-GPU).

License

This model is a derivative of Llama-3.1-Nemotron-Nano-8B-v1 and is governed by the NVIDIA Open Model License and the Llama 3.1 Community License Agreement ("Built with Llama"). See LICENSE, LICENSE.llama, and NOTICE. You must retain attribution and the notice that the weights were modified (quantized).

BareMetalAI/Llama-3.1-Nemotron-Nano-8B-W4A16-AWQ

Llama-3.1-Nemotron-Nano-8B W4A16-AWQ (int4)

Quantization

Hardware

License

BareMetalAI/
Llama-3.1-Nemotron-Nano-8B-W4A16-AWQ