You can deploy this model and start earning money today!

yapwithai/
orpheus-3b-tts-trt-awq

Name: orpheus-3b-tts-trt-awq
Availability: InStock

yapwithai/canopy-orpheus-3b-0.1-ft

finetune

apache-2.0

Views

No views yet

Orpheus 3B — INT4 AWQ Quantized

Streaming optimized Orpheus 3B quant. Meant to run on NVIDIA A100 with TensorRT-LLM

Model Overview

This is a streaming optimized INT4 AWQ quantized version of yapwithai/canopy-orpheus-3b-0.1-ft, meant to run with TensorRT-LLM.

Key Features:

Optimized for Production: Built for high-throughput, low-latency TTS serving
Faster Inference: Up to ~3x faster than FP16 with minimal perceived quality loss
Memory Efficient: ≈4x smaller weights vs. FP16 (INT4)
Ready for Streaming: Designed for real-time streaming TTS backends
Calibrated: Calibrated for 60 tokens of input and up to 1162 tokens of output (roughly 14 seconds worth of audio per text chunk)

Technical Specifications

Specification	Details
Source Model	yapwithai/canopy-orpheus-3b-0.1-ft
Quantization Method	AWQ (Activation-aware Weight Quantization)
Precision	INT4 weights, INT8 KV cache
AWQ Group/Block Size	128
TensorRT-LLM Version	`1.2.0rc5`
Max Batch Size	16
SM Arch	sm80
GPU Name	NVIDIA A100
GPU VRAM	80 GB
CUDA Toolkit	13.0
NVIDIA Driver	550.127.05
Generated	2025-12-22

Artifact Layout

trt-llm/
  checkpoints/               # Quantized TRT-LLM checkpoints (portable)
    *.safetensors
    config.json
  engines/sm80_trt-llm-1.2.0rc5_cuda13.0/  # Built TensorRT-LLM engines (hardware-specific)
    rank*.engine
    build_metadata.json
    build_command.sh

Quantization Details

Method: Activation-aware weight quantization (AWQ)
Calibration size: 256
AWQ block/group size: 128
DType for build: float16

Configuration Summary

json

{
  "quantization": {
    "weights_precision": "int4_awq",
    "kv_cache_dtype": "int8",
    "awq_block_size": 128,
    "calib_size": 32
  },
  "build": {
    "dtype": "float16",
    "max_input_len": 60,
    "max_output_len": 1162,
    "max_batch_size": 16,
    "engine_label": "sm80_trt-llm-1.2.0rc5_cuda13.0",
    "tensorrt_llm_version": "1.2.0rc5"
  },
  "environment": {
    "sm_arch": "sm80",
    "gpu_name": "NVIDIA A100",
    "cuda_toolkit": "13.0",
    "nvidia_driver": "550.127.05"
  }
}

Use Cases

Realtime Voice: assistants, product demos, interactive agents
High-throughput Serving: batch TTS pipelines, APIs
Edge & Cost-sensitive: limited VRAM environments

Advanced Configuration

Max input length: tune --max_input_len
Max output length: tune --max_seq_len
Batch size: tune --max_batch_size
Plugins: --gpt_attention_plugin, --context_fmha, --kv_cache_type paged

Requirements & Compatibility

System Requirements

GPU: NVIDIA, Compute Capability ≥ 8.0 (A100/RTX 40/H100 class recommended)
CUDA: 13.0
Python: 3.10+

Framework Compatibility

TensorRT-LLM (engines), version 1.2.0rc5
TRT-LLM Checkpoints are portable across systems; engines are not

Installation

bash

pip install huggingface_hub
# Install TensorRT-LLM per NVIDIA docs
# https://nvidia.github.io/TensorRT-LLM/

Troubleshooting

Engine not portable

Engines are specific to GPU SM and TRT/CUDA versions. Rebuild on the target system or download a matching `engines/sm80_trt-llm-1.2.0rc5_cuda13.0` variant if provided.

OOM or Slow Loading

Reduce `max_seq_len`, lower `max_batch_size`, and ensure `gpu_memory_utilization` on your server is tuned to your GPU.

Additional Resources

TensorRT-LLM Docs: https://nvidia.github.io/TensorRT-LLM/
Activation-aware Weight Quantization (AWQ): https://github.com/mit-han-lab/llm-awq

License

This quantized model inherits the license from the original model: Apache 2.0

yapwithai/orpheus-3b-tts-trt-awq