You can deploy this model and start earning money today!

D3vil-Reap3r/
sarvam-1-int4-awq

Name: sarvam-1-int4-awq
Availability: InStock

llama

apache-2.0

Views

No views yet

Sarvam-1 INT4-AWQ for TensorRT-LLM

Model Description

This is a quantized version of the sarvamai/sarvam-1 2B parameter model. It has been optimized for high-performance inference on memory-constrained edge hardware.

Quantization Details

Method: W4A16 AWQ (Activation-aware Weight Quantization)
Tool: NVIDIA ModelOpt
Precision: INT4 Weights, FP16 Activations

Performance Benchmarks (NVIDIA RTX 3050 4GB)

Using the TensorRT-LLM C++ Executor API:

Throughput: 86.77 tokens/sec
TTFT: 22.75 ms
VRAM Footprint: 3.03 GB (Strictly Fenced)

Usage

This model can be compiled into a TensorRT engine using trtllm-build.

bash

trtllm-build --checkpoint_dir . \
             --output_dir ./engine \
             --gemm_plugin float16 \
             --max_batch_size 1 \
             --max_input_len 512 \
             --max_seq_len 1024

Disclaimer

This model is a derivative of the Sarvam-1 model. Please refer to the original license for usage restrictions.

D3vil-Reap3r/sarvam-1-int4-awq