Speech recognition in two .pte files: the encoder runs once per 30-second window, the
decoder once per generated token. Putting them in one graph would re-encode the audio on
every step.
graph
build
file
size (MB)
corr vs fp32 eager
ms
eager ms
encoder
XNNPACK fp32
whisper_large_v3_turbo_encoder_xnnpack_fp32.pte
2548.3
1.000000
1800.1
860.5
encoder
XNNPACK int8
whisper_large_v3_turbo_encoder_xnnpack_int8.pte
662.4
0.996407
1582.1
866.5
encoder
Core ML
whisper_large_v3_turbo_encoder_coreml_all.pte
1275.7
0.998309
589.1
869.7
decoder
XNNPACK fp32
whisper_large_v3_turbo_decoder_xnnpack_fp32.pte
953.3
1.000000
56.6
43.2
decoder
XNNPACK int8
whisper_large_v3_turbo_decoder_xnnpack_int8.pte
440.1
0.998471
46.7
40.7
Every file takes and returns fp32 tensors (token ids stay int64), so any encoder pairs with
any decoder. The lightest working pair is 1102.5 MB.
Encoder input: log-mel spectrogram [1, 128, 3000] — 30 s at 16 kHz, 128 mel bins, hop
160, window 400, exactly what WhisperFeatureExtractor produces
Decoder input: the encoder output plus decoder_input_ids [1, 128] int64,
left-aligned and padded. Start with <|startoftranscript|>, a language token,
<|transcribe|>, <|notimestamps|>.
Decoding
No KV cache: the decoder is a static graph over a fixed 128-token window, so a greedy step
is take argmax of row len-1, append it, run again. Stop at <|endoftext|> (50257). 128
tokens covers a 30-second window of ordinary speech; past that, start a new window.
That costs a full 128-position forward pass per token, which is the price of a static graph
that runs unchanged across runtimes and precisions.
The two wrappers compose back to WhisperForConditionalGeneration exactly — max_abs_diff
0.000e+00 — and every graph matches torch fp32 eager at the correlations above. Timings
are medians over 5 runs in one process: a relative reference, not a device number.
Two things worth knowing about the sizes
The decoder .pte is larger than the decoder's weights. Whisper ties proj_out.weight
to decoder.embed_tokens.weight, but the two uses need different representations: an
embedding table the portable kernels index into, and the same values packed into the XNNPACK
delegate's blob for the output matmul. Tying them in PyTorch does not tie them here.
Referencing the weight through F.linear instead of the proj_out module does not either —
exported both ways, whisper-tiny's decoder comes out at 198.0 MB exactly.
The decoder's int8 build is the smallest portable one. Dynamic int8 quantizes the linear weights and
leaves the token embedding table in fp32, and that table is 265.5 MB — 51,865 tokens at
1280 dimensions. There is no fp16 build at this size to compare against: int8 comes in at 440.1 MB and is the smallest portable decoder here.
Until recently there was no decoder int8 build at all, and this card said PT2E was observing
the int64 decoder_input_ids. That was wrong on both halves.
XNNPACKQuantizer.transform_for_annotation rewrites every scalar argument of
add.Tensor/mul.Tensor as torch.tensor(float(arg)) whatever the node's dtype — one line
in ExecuTorch's backends/xnnpack/quantizer/xnnpack_quantizer_utils.py, still present on
main. In this decoder the casualty is position_ids = torch.arange(...) + past_key_values_length
(modeling_whisper.py:749, past_key_values_length being a python int): it comes back
float32, and the failure lands on self.weight[position_ids] — the position embedding
lookup, not the token ids, and no observer involved. Measured by running prepare_pt2e
with an empty quantizer and printing the failing node.
The decoder's Core ML build is not published. It converts, at 343.4 MB and correlation 0.961553 against fp32 eager, and then fails the only test that matters here: measured end to end — word error rate against the fp32 decoder: mean WER 43.9% (worst clip 66.7%) over 5 spoken sentences, the coreml_all decoder against the fp32 one with the other half and the waveform held identical; the fp32 arm transcribes all five correctly, so the comparison is against a working control rather than against noise.
Checked in the task's own units
Correlation is a first filter. These are the numbers that decide:
encoder int8 — measured end to end — word error rate against the fp32 encoder: mean WER 0.0% (worst clip 0.0%) over 5 spoken sentences, the int8 encoder against the fp32 one with the other half and the waveform held identical; the fp32 arm transcribes all five correctly, so the comparison is against a working control rather than against noise.
decoder int8 — measured end to end — word error rate against the fp32 decoder: mean WER 0.0% (worst clip 0.0%) over 5 spoken sentences, the int8 decoder against the fp32 one with the other half and the waveform held identical; the fp32 arm transcribes all five correctly, so the comparison is against a working control rather than against noise.
The sensitivity of that test, measured by injecting random noise into whisper-tiny's encoder output: rel_l2 0.03 (what int8 actually costs) and 0.10 both give WER 0.000; 0.20 and 0.40 give 0.025. Five clean sentences leave headroom, so a pass means does not break the transcript, not indistinguishable at any error level.
Conversion
python convert/export_whisper.py large_v3_turbo
The ExecuTorch tree ships a single-graph Whisper example under examples/models/whisper;
this is that model with the halves separated.
