Views
No views yet
convert_lm_to_gguf.py
(s = 1/mean(|W|); W' = round(clamp(W·s, −1, 1))/s) is applied to the q/k/v/o/gate/up/down
projections at export, matching the official GGUF deployment bit-for-bit. Ternary×scale values
are exactly representable in fp16 and in blockwise int4 (MatMulNBits, block 32, symmetric),
so every tier decodes the same ternary model — tiers differ only in storage and kernels.
Activations run in float (the ggml runtime quantizes activations to int8; float is equal or
better).| Tier | Graphs | Notes |
|---|---|---|
| fp32 | audio_encoder.onnx (+.onnx_data), decoder_init.onnx, decoder_step.onnx, decoder_weights.data | parity reference (~10.4 GB) |
| fp16 | *.fp16.onnx, decoder_weights.fp16.data | token-EXACT vs reference (~5.4 GB) |
| int4 | *.int4.onnx, decoder_weights.int4.data | native BitNet tier, token-EXACT (~3.5 GB) |
com.microsoft::MatMulNBits
(q = w/s + 8 ∈ {7,8,9}, block 32, per-block scale = the ternary scale — bit-exact packing,
verified tensor-by-tensor at build and token-exact end-to-end); lm_head/embeddings stay fp16.
There is intentionally no int8 tier: dynamic-activation int8 measurably degraded
transcripts while being larger than the exact int4.embed_tokens.bin (raw fp16 [151936 × 1536] row-major token-embedding
table for host-side lookup), tokenizer.json (Qwen2.5), config.json.audio_encoder(input_values[1,T] @24kHz) -> audio_features[1,N,1536] # N = ceil(T/3200)
decoder_init(input_ids[1,S], position_ids[1,S],
audio_features[1,N,1536], audio_offset[1])
-> logits[1,1,V], present_keys/values [28,1,2,S,128]
decoder_step(input_embeds[1,1,1536], position_ids[1,1],
past_keys/values [28,1,2,P,128])
-> logits[1,1,V], present_keys/values [28,1,2,P+1,128]decoder_init owns the embedding table and splices audio_features over the
<|speech_pad|> (151648) positions starting at audio_offset. Prompt template
(prompt_builder.h "text" format, no generation prompt — the model emits the
<|im_start|>assistant turn itself):<|im_start|>system\nYou are a helpful assistant that transcribes audio input into text output in JSON format.<|im_end|>\n
<|im_start|>user\n<|speech_start|>{<|speech_pad|> × N}<|speech_end|>\nThis is a {D:.2f} seconds audio, please transcribe it.<|im_end|>\n\nThis is a {D:.2f} seconds audio, with extra info: {context}\n\nPlease transcribe it.
Stop tokens: <|im_end|> (151645), <|endoftext|> (151643). Greedy decode.microsoft/VibeVoice-ASR-BitNet fp32 QAT master safetensors.transformers convert_vibevoice_asr_to_hf.py (v5.14 key layout),
ternarization per VibeASR.cpp, torch.onnx.export (TorchScript exporter, opset 17).