Views
No views yet
| Parameter | Value |
|---|---|
| Method | AWQ (Activation-aware Weight Quantization) |
| Weight Precision | 4-bit |
| Activation Precision | 16-bit |
| Format | compressed-tensors |
| Quantization Tool | llmcompressor |
| Model Size Reduction | ~75% |
| Calibration Samples | 512 |
| Calibration Dataset | snorbyte/indic-tts-sample-snac-encoded |
| Metric | Original Model | This Model (AWQ) |
|---|---|---|
| Model Size | ~8GB | ~3.5GB (60% reduction) |
| Inference Speed | Baseline | Faster (4-bit computation) |
| Memory Usage | High | Low |
| Audio Quality | Reference | Minimal degradation |
1docker run \
2--runtime nvidia \
3--gpus all \
4-v ~/.cache/huggingface:/root/.cache/huggingface \
5-v ~/.cache/vllm:/root/.cache/vllm \
6-v ~/snor-quant:/models \
7-p 8002:8002 \
8--env "HF_HUB_ENABLE_HF_TRANSFER=1" \
9--env "HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}" \
10--env "HF_HUB_OFFLINE=1" \
11--ipc=host \
12--shm-size 32g \
13--log-opt max-size=10m \
14--log-opt max-file=3 \
15vllm/vllm-openai:latest \
16--port 8002 \
17--model "/models/snorTTS-Indic-v0-AWQ-W4A16" \
18--served-model-name llm \
19--host 0.0.0.0 \
20--max-model-len 2048 \
21--max-num-seqs 5 \
22--gpu-memory-utilization 0.20 \
23--dtype auto \
24--quantization compressed-tensors \
25--trust-remote-code \
26--uvicorn-log-level info