You can deploy this model and start earning money today!

smajji/
qwen3-tts-customvoice-edgellm-aarch64

Name: qwen3-tts-customvoice-edgellm-aarch64
Availability: InStock

Views

No views yet

Qwen3-TTS-12Hz-0.6B-CustomVoice — TensorRT-Edge-LLM engines (Jetson aarch64)

TensorRT engines (aarch64, Jetson Orin SM87, TensorRT 10.13.1 / CUDA 12.1) for Qwen3-TTS-12Hz-0.6B-CustomVoice, built with the ai.core.tts fork runtime's 0.8-era toolchain (edgellm_version 0.8.0, contiguous KV). Each engine dir is self-contained: llm.engine/sidecars (*.safetensors) + config.json + tokenizer files — deploy by copy.

Quality note (measured 2026-08-19): FP16 is the production baseline. Calibrated INT4 (uniform 196-GEMM or mixed 168-GEMM) removed the old INT4 noise/crackle but keeps a pacing/hesitation gap vs FP16 on longer text; short utterances ≈ FP16. INT4 RSS at 512/512 ≈ FP16 (embeddings dominate), so INT4's memory win needs long sequence lengths.

Layout

engines/
  talker_fp16/  code_predictor_fp16/        <- FP16 talker+CP (PRODUCTION) @512/512
  code2wav_base/                            <- Base-model vocoder (16 quantizers, 48kHz)
  talker_int4_legacy/ code_predictor_int4_legacy/        <- legacy INT4 (fallback, 196 GEMM)
  talker_int4_calibrated/  code_predictor_int4_calibrated/ <- N4 talker-shaped AWQ INT4 @512
  talker_int4_mixed/  code_predictor_int4_mixed/          <- layers 0/1/26/27 FP16 + 2-25 INT4 @512
code/
  ai.core.tts-src.tar.gz   <- full runtime/server source + progress.md + tools (fork, feat/qwen3-custom-voice)
  patches_n3/              <- toolchain shim, lm_head patch, engine build/graft scripts
  gradio_tts.py            <- browser control panel (runs on any machine, pure-python client)
manifest.json              <- sha256 of every engine file

Serve (on the Jetson)

bash

cd /mnt/vblkdev2/smajji/ai.core.ttsv2/ai.core.tts   # ai.core.tts deploy tree
source setup_trt.sh
export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH
export EDGELLM_PLUGIN_PATH=trt-edge-llm/prebuilt/libNvInfer_edgellm_plugin.so
export TMPDIR=/mnt/vblkdev2/smajji/tmp

./build/tts_inference \
  <path>/engines/talker_fp16 \
  <path>/engines/code_predictor_fp16 \
  <path>/engines/code2wav_base \
  assets 8890 --model-type customvoice
# then: asr_client <jetson-ip> 8890 request <id> English af_heart.bin "<text>" --speaker-name serena

Speakers: serena ryan vivian aiden eric dylan sohee ono_anna uncle_fu. Options per request: --instruct "<style>", --first-chunk-frames N, --chunk-frames N, --non-streaming, --repetition-penalty F (TTFA ~145-150 ms at first-chunk=1).

Provenance

Exported from Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice (HF); talker=LLM(codec), code_predictor, code2wav(Base).
Engines built on-device (aarch64) with the 0.8-era toolchain; calibration texts numeral/address-heavy.
Full history + A/B evidence: code/ai.core.tts-src.tar.gz → progress.md (§3.25–3.27, §4).
Integrity: manifest.json (sha256 per file).

FP8-embedding + INT4-G32/G64 variants (added 2026-08-25)

All talker variants now also ship as *_fp8emb engine dirs: the text_embedding.safetensors inside is the FP8 E4M3 (group-128, amax/448) sidecar (321 MB vs 622 MB, −291.7 MB VRAM/server, audibly lossless). The fork runtime loads FP8 natively (no flag). Original 622 MB FP16 tables remain in the plain talker_fp16 / talker_int4_* dirs.

INT4 group_size improvements (quality, same 2.0 GB): the plugin now supports G=32/64/128 (the repo ships the G32-capable trt-edge-llm/prebuilt/libNvInfer_edgellm_plugin.so).

Dir	Stack	Typical VRAM (jtop)	Notes
`talker_fp16_fp8emb`	FP16 + FP8 emb	~2.6 G	Production-quality voice
`talker_int4_g32_fp8emb`	INT4-G32 + FP8 emb	~2.0 G	Best INT4 — closest to FP16 pacing; see progress §3.33
`talker_int4_g64_fp8emb`	INT4-G64 + FP8 emb	~2.0 G	Mid-step
`talker_int4_mixed_fp8emb`	INT4-mixed + FP8 emb	~2.0 G	Reference

A/B (2026-08-25, nav text A): FP16 21.0 s/17.4 % silence → G128 30.0/26.4 → G64 27.2/23.6 → G32 26.2/19.8. All *_fp8emb variants: 0 noise windows, deterministic. Full log: docs/progress.md, progress.md §3.29–§3.33, build.md.

Merging into an existing assets/ tree: copy the engine dirs into assets/Qwen3-TTS-12Hz-0.6B-CV/ (or point launch args at <repo>/engines/…) and launch tts_inference <talker_dir> <code_predictor_dir> <code2wav_dir> assets <port> --model-type customvoice. G32/G64 engines require the repo's G32-capable plugin (already prebuilt); keep 3 servers max.

smajji/qwen3-tts-customvoice-edgellm-aarch64