High-Performance Inference: Optimized for NVIDIA GPUs with TensorRT-LLM
Memory Efficient: 4-bit weights reduce VRAM usage vs FP16
Production Ready: Built for low-latency, high-throughput chat serving
Portable Checkpoints: Checkpoints work across systems; engines are hardware-specific

Technical Specifications

Specification	Details
Source Model	https://huggingface.co/ReadyArt/Broken-Tutu-24B-Transgression-v2.0
Quantization Method	INT4-AWQ
Precision	4-bit weights
KV Cache	int8
KV Cache Type	paged
KV Reuse	enabled
Block/Group Size	128
TensorRT-LLM Version	`1.2.0rc5` (used for quantization)
Max Batch Size	64
Max Input Length	5525
Max Output Length	150
SM Architecture	sm90
GPU	NVIDIA H100 NVL
CUDA Toolkit	13.0
Generated	2026-01-03 17:53:47 UTC

Artifact Layout

trt-llm/
  checkpoints/
    *.safetensors
    config.json
  engines/sm90_trt-llm-1.2.0rc5_cuda13.0/ 
    rank*.engine
    config.json

Checkpoints: Portable across systems with compatible TensorRT-LLM versions; rebuild engines on the target GPU
Engines: Hardware-specific (rebuild for different GPU/CUDA versions/SMs, e.g., H100/H200/B200/Blackwell, L40S, 4090/RTX)
INT4-AWQ checkpoints are portable across sm89/sm90+ GPUs; rebuild engines for the target GPU (e.g., H100/H200/B200/Blackwell, L40S, 4090/RTX) or reuse one of the prebuild engines in case they match your GPU.

Engine fails to load on different GPU

Engines are compiled for specific SM architecture and CUDA version. Either:

Out of Memory

Reduce `max_batch_size` or `max_seq_len` when building the engine. Adjust `kv_cache_config.free_gpu_memory_fraction` at runtime.

This quantized model inherits the license from the original base model: apache-2.0