ai.core.tts runs on this device.engines-071/ — FP16 TRT engines:
engines-071/llm/ (llm.engine, config.json, tokenizer, embedding.safetensors)
and engines-071/audio/audio/ (audio encoder engine — note the nested
audio/audio/ layout).onnx-071/ — 0.7.1-format FP16 ONNX exports (llm/ + audio/).engines-int4-qad/, engines-int4-vanilla/ — INT4 engines (W4A16
AWQ, group 128, lm_head FP16): each has llm/ only; the audio engine is
reused from engines-071/audio (ln -sfn engines-071/audio <root>/audio).
qad = vrfai/Qwen3-ASR-0.6B-int4-QAD weights (distilled+AWQ).vanilla = fresh ModelOpt int4_awq of the base checkpoint.engines-int4-qad-kv1024/, engines-int4-vanilla-kv1024/ — the
memory-tuned INT4 engines (recommended): built with
--maxInputLen 896 --maxKVCacheCapacity 1024 (~65 s audio/utterance;
KV 470 MB → 117 MB) plus FP8 embedding (embedding.safetensors is E4M3
embedding_scale; 311 MB → 156 MB, runtime dequantizes on the fly).
Resident estimate ≈ 1.4 GB vs ~2 GB for the 4096/FP16 build.onnx-071-int4-qad/, onnx-071-int4-vanilla/ — the INT4 ONNX
exports that built those engines (lm_head excluded → fp16 MatMul; the V3
plugin-output→Cast issue that blocked TRT 10.13 builds is thereby avoided).bin-aarch64/ — the three executables built on the device, plus
libedgellmCore.a and libNvInfer_edgellm_plugin.so (aarch64):
qwen3_asr_trt (standalone CLI), android_asr_server (TCP ASR server),
voice_streaming (streaming server).code/ — full ai.core.asr source with the vendored 0.7.1 runtime +
export/build scripts + docs/int4-run.md + PROGRESS docs.test0.wav — 6.6 s English sample used for validation.1export LD_LIBRARY_PATH=/usr/local/TensorRT-10.13/lib:/usr/local/TensorRT-10.13/targets/aarch64-linux-gnu/lib:/usr/local/cuda-12.1/lib64:/usr/local/lib
2export EDGELLM_PLUGIN_PATH=/path/to/bin-aarch64/libNvInfer_edgellm_plugin.so # must match engine era
3export TMPDIR=/media/sda1/qwen3-asr/tmp
4
5# FP16 standalone
6./bin-aarch64/qwen3_asr_trt --engine-dir engines-071 test0.wav
7
8# INT4 standalone (attach the shared audio engine first)
9ln -sfn engines-071/audio engines-int4-vanilla/audio
10ln -sfn engines-071/audio engines-int4-qad/audio
11./bin-aarch64/qwen3_asr_trt --engine-dir engines-int4-vanilla test0.wav
12./bin-aarch64/qwen3_asr_trt --engine-dir engines-int4-qad test0.wav
13
14# Memory-tuned INT4 (KV 1024 + FP8 embedding) — recommended
15ln -sfn engines-071/audio engines-int4-vanilla-kv1024/audio
16ln -sfn engines-071/audio engines-int4-qad-kv1024/audio
17./bin-aarch64/qwen3_asr_trt --engine-dir engines-int4-vanilla-kv1024 test0.wav
18./bin-aarch64/qwen3_asr_trt --engine-dir engines-int4-qad-kv1024 test0.wav
19
20# Android ASR TCP server (int16 16kHz PCM protocol)
21./bin-aarch64/android_asr_server --engine qwen3asr --model-dir engines-int4-vanilla-kv1024 --port 9092
22
23# voice_streaming server (models dir = cwd/models -> symlink the engine root)
24mkdir -p vs-cwd && ln -sfn engines-int4-qad-kv1024 vs-cwd/models
25./bin-aarch64/voice_streaming 9121 --load=qwen3asr # or 'set_model_qwen3asr <engine-root>' over the socket"language EnglishAfter early nightfall, the yellow lamps would light up here and there—the squalid quarter of the brothels."
(identical for FP16 and both INT4 variants; INT4 decode ≈ 130 ms vs FP16 ≈ 229 ms).input_tokens in the standalone JSON is always 0 (0.7.1 responses expose
no input-token count); the "language English" prefix is the model's
language-tag convention..safetensors under $TMPDIR or
/dev/shm.kALIASED_PLUGIN_IO_10_03
builder feature): see code/docs/int4-run.md and code/PROGRESS.md.