You can deploy this model and start earning money today!

rungalileo/
mistral-7b-instruct-v0.3-trtllm-ckpt-wq_fp8-kv_fp8

Name: mistral-7b-instruct-v0.3-trtllm-ckpt-wq_fp8-kv_fp8
Availability: InStock

tensorrt-llm

fp8

kv-cache-quantization

text-generation

mistral

mistralai/Mistral-7B-Instruct-v0.3

finetune

apache-2.0

Views

No views yet

Mistral-7B-Instruct-v0.3 TensorRT-LLM checkpoint (FP8 weight + FP8 KV)

TensorRT-LLM checkpoint for Mistral-7B-Instruct-v0.3, with FP8 (W8A8) weight quantization and FP8 KV cache. Use with trtllm-build to produce an engine for inference.

Model details

Item	Value
Base model	Mistral-7B-Instruct-v0.3
Framework	TensorRT-LLM (checkpoint format)
Weight quantization	FP8 (W8A8)
KV cache	FP8
Producer	TensorRT-Model-Optimizer llm_ptq + TensorRT-LLM convert_checkpoint (--use_fp8, --fp8_kv_cache)
Architecture	MistralForCausalLM (decoder-only)

Build (how to produce this checkpoint)

FP8 requires a two-step pipeline: (1) run Model Optimizer llm_ptq to quantize the Hugging Face model to FP8; (2) run TensorRT-LLM convert_checkpoint with the PTQ output to produce this checkpoint.

1. Environment and dependencies

bash

sudo apt install git-lfs
git lfs install

pip install tensorrt_llm --extra-index-url https://pypi.nvidia.com
# Install TensorRT-Model-Optimizer (required for FP8 quantization)
# See https://github.com/NVIDIA/TensorRT-Model-Optimizer

2. Quantize base model to FP8 (llm_ptq)

Clone the base model and run Model Optimizer's llm_ptq to produce an FP8-quantized HF-format directory. Then run TensorRT-LLM convert_checkpoint (Mistral example):

bash

# Example: after llm_ptq has produced PTQ output directory,
# run convert_checkpoint with that directory as --model_dir:
python TensorRT-LLM/examples/mistral/convert_checkpoint.py \
  --model_dir ./path/to/ptq_output \
  --output_dir ./mistral-7b-instruct-v0.3-trtllm-ckpt-wq_fp8-kv_fp8 \
  --dtype float16 \
  --use_fp8 \
  --fp8_kv_cache

(Without --use_fp8, the engine can produce NaN logits; both flags are required for correct FP8 weight + FP8 KV.)

3. Output

After conversion, --output_dir contains config.json and rank0.safetensors; that is the checkpoint in this repo.

Upload (how to upload to Hugging Face)

bash

cd ./mistral-7b-instruct-v0.3-trtllm-ckpt-wq_fp8-kv_fp8

huggingface-cli repo create rungalileo/mistral-7b-instruct-v0.3-trtllm-ckpt-wq_fp8-kv_fp8 --repo-type model
huggingface-cli upload rungalileo/mistral-7b-instruct-v0.3-trtllm-ckpt-wq_fp8-kv_fp8 . --repo-type model

How to use

1. Build engine

Requires TensorRT-LLM and tensorrt_llm installed:

bash

git clone https://huggingface.co/rungalileo/mistral-7b-instruct-v0.3-trtllm-ckpt-wq_fp8-kv_fp8
cd mistral-7b-instruct-v0.3-trtllm-ckpt-wq_fp8-kv_fp8

trtllm-build --checkpoint_dir . --output_dir ./engine \
  --max_batch_size 1 --max_input_len 512 --max_seq_len 1024

2. Run inference

Use a tokenizer from the base model (e.g. mistralai/Mistral-7B-Instruct-v0.3):

bash

trtllm-serve ./engine --tokenizer mistralai/Mistral-7B-Instruct-v0.3 --port 8000
# OpenAI-compatible API: http://localhost:8000/v1/completions

Files in this repo

config.json – TensorRT-LLM model config
rank0.safetensors – Rank 0 weights (single-GPU)

rungalileo/mistral-7b-instruct-v0.3-trtllm-ckpt-wq_fp8-kv_fp8

Mistral-7B-Instruct-v0.3 TensorRT-LLM checkpoint (FP8 weight + FP8 KV)

Model details

Build (how to produce this checkpoint)

1. Environment and dependencies

2. Quantize base model to FP8 (llm_ptq)

3. Output

Upload (how to upload to Hugging Face)

How to use

1. Build engine

2. Run inference

Files in this repo

References

rungalileo/
mistral-7b-instruct-v0.3-trtllm-ckpt-wq_fp8-kv_fp8