Views
No views yet
.pte file
for Parakeet TDT 0.6B with
CUDA-Windows backend (NVIDIA GPU), bf16 precision, 4-bit weight linear
quantization, and 8-bit weight embedding quantization.| Component | Quantization | Precision |
|---|---|---|
| Encoder linear layers | 4-bit weight-only (4w, group_size=32) | int4 weights, bf16 activations |
| Decoder linear layers | 4-bit weight-only (4w, group_size=32) | int4 weights, bf16 activations |
| Decoder embedding | 8-bit weight-only (8w) | int8 |
| Conv / LSTM / Norm | None | bf16 |
| Preprocessor | None | fp32 (always CPU) |
1git clone https://github.com/pytorch/executorch/ ~/executorch
2cd ~/executorch && pip install .1cmake --workflow --preset llm-release-cuda
2Push-Location examples/models/parakeet
3cmake --workflow --preset parakeet-cuda
4Pop-Location1pip install huggingface_hub
2huggingface-cli download younghan-meta/Parakeet-TDT-ExecuTorch-CUDA-Windows-Quantized --local-dir parakeet_cuda_windows_quantized1.\cmake-out\examples\models\parakeet\Release\parakeet_runner.exe `
2 --model_path parakeet_cuda_windows_quantized\model.pte `
3 --data_path parakeet_cuda_windows_quantized\aoti_cuda_blob.ptd `
4 --audio_path C:\path\to\audio.wav `
5 --tokenizer_path parakeet_cuda_windows_quantized\tokenizer.model--timestamps segment — timestamp granularity: none|token|word|segment|all (default: segment)1pip install "nemo_toolkit[asr]"
2
3python examples/models/parakeet/export_parakeet_tdt.py \
4 --backend cuda-windows \
5 --dtype bf16 \
6 --qlinear_encoder 4w \
7 --qlinear_encoder_group_size 32 \
8 --qlinear 4w \
9 --qlinear_group_size 32 \
10 --qembedding 8w \
11 --output-dir ./parakeet_cuda_windows_quantizedx86_64-w64-mingw32-g++ on PATH and WINDOWS_CUDA_HOME
pointing to the extracted Windows CUDA package. See the
Parakeet README
for detailed setup steps.tile_packed_to_4d packing format is not supported for cuda-windows
(the cross-compilation path keeps tensors on CPU during export and
aten::_convert_weight_to_int4pack only has a CUDA kernel). The quantization
uses the generic IntxWeightOnlyConfig path instead.| Metric | Quantized | vs Non-Quantized |
|---|---|---|
| Prefill throughput | 3,218 tok/s | 2.5× faster |
| Decode throughput | 1,545 tok/s | 1.2× faster |
| Model load time | 1.6 s | 2.5× faster |
| Time to first token | 86 ms | 2.4× faster |
| Total inference | 144 ms | 1.9× faster |
| Real-time factor | 139× real-time | — |
| Model size | 763 MB | 3.2× smaller |
| Transcription accuracy | Identical words | No loss |