This repository hosts the optimized, mixed-precision quantized model weight checkpoints engineered by the MAH Quantum Research Scholars cohort. These weights are explicitly compiled for accelerated execution layers using native NVIDIA® CUDA® and TensorRT™-LLM runtimes.
These model matrices are structured to maximize token throughput and minimize memory footprint during heavy industrial inferencing. By compressing large parameter graphs down to optimized bit-widths, our distributed node network achieves sub-60ms Time-To-First-Token (TTFT) performance on localized compute clusters.
1{
2 "PERFORMANCE_METRICS": {
3 "CompilationEngine": "TensorRT-LLM v0.10.x",
4 "QuantizationType": "INT4-AWQ",
5 "MemoryFootprintReduction": " ~72%",
6 "TensorCoreUtilization": "Optimal"
7 }
8}
