Whisper ACFT — short-window TFLite (LiteRT), tiny/base/small × multilingual/.en
TFLite (LiteRT) exports of the FUTO
audio-context fine-tuned (ACFT) Whisper checkpoints: fixed
5 s / 10 s / 30 s window graphs split into
encode and
decode signatures, matching the graph interface of
litert-community/whisper-tiny and
litert-community/whisper-base. Six models in one repo:
tiny,
base,
small (multilingual) and
tiny.en,
base.en,
small.en (English-only), each in three window lengths.
No retraining — the weights are FUTO's Apache-2.0 ACFT checkpoints (
futo-org/whisper-acft method and training; HF checkpoints
futo-org/acft-whisper-{tiny,base,small}{,.en}). This repo only converts them: transformers
TFWhisperForConditionalGeneration → two-signature graph → converter-time dynamic-range quantization (int8 weights, fp32 activations,
tf.lite.Optimize.DEFAULT). Conversion by the
LiteRT-LM-Unity project.
Why short windows
Whisper always encodes a 30 s window; for sub-5 s voice commands most encoder work is spent on padding. The encoder conv stack is length-agnostic and the encoder positional table is a fixed sinusoid, so slicing the first N rows of embed_positions yields a shorter window — exactly what whisper.cpp's audio_ctx parameter does at runtime. Stock Whisper decoders are trained on the full 30 s window, so shrinking it puts them out of distribution (repetition loops: stock base at a 5 s window measured CER 10.4 on a sentence clip in our evaluation); the ACFT checkpoints are fine-tuned via self-distillation (L2 on hidden states) to tolerate exactly this.
Repo layout
One folder per model; filenames encode model and window. All files are dynamic-range quantized (int8 weights, fp32 activations).
| Folder | Source checkpoint | Files (5s / 10s / 30s) | Size |
|---|
tiny/ | futo-org/acft-whisper-tiny | acft_whisper_tiny_{5s,10s,30s}_drq.tflite | 57–58 MB |
base/ | futo-org/acft-whisper-base | acft_whisper_base_{5s,10s,30s}_drq.tflite | 97–99 MB |
small/ | futo-org/acft-whisper-small | acft_whisper_small_{5s,10s,30s}_drq.tflite | 273–276 MB |
tiny.en/ | futo-org/acft-whisper-tiny.en | acft_whisper_tiny.en_{5s,10s,30s}_drq.tflite | 57–58 MB |
base.en/ | futo-org/acft-whisper-base.en | acft_whisper_base.en_{5s,10s,30s}_drq.tflite | 97–99 MB |
small.en/ | futo-org/acft-whisper-small.en | acft_whisper_small.en_{5s,10s,30s}_drq.tflite | 273–276 MB |
Window ↔ encode input shape: 5 s = [1, 80, 500], 10 s = [1, 80, 1000], 30 s = [1, 80, 3000].
Integration notes
- Fixed short-window graphs: the encode signature input is
[1, 80, frames] with frames = 500 / 1000 / 3000. The runtime must size the mel window from the encode signature's input shape (compute/pad the log-mel to exactly that frame count) instead of assuming 3000 frames.
- The decode signature input order is
(mask, audio, tokens) — opposite of the stock litert-community exports — bind decode tensors by shape/name rather than position. The decoder is a fixed-length full re-run per step (no KV cache), matching the tiny/base graph interface.
- Reference implementation: the JNI bridge in LiteRT-LM-Unity, which auto-detects 500/1000/3000-frame windows from the encode signature on one code path.
Token ids — multilingual vs .en
- Multilingual (
tiny/, base/, small/): 80 mel bins / vocab 51865 — classic Whisper token layout, same as litert-community tiny/base. Decode prompt e.g. [<|startoftranscript|>=50258, <|lang|>, <|transcribe|>=50359, <|notimestamps|>=50363]. Tokenizer: tokenizer.json from the matching openai/whisper-* repo.
- English-only (
tiny.en/, base.en/, small.en/): 80 mel bins / vocab 51864 — English-only layout: <|startoftranscript|> = 50257, <|endoftext|> = 50256, <|notimestamps|> = 50362; there are no language/task tokens — the decode prompt is [50257, 50362]. Tokenizer: tokenizer.json from the matching openai/whisper-*.en repo.
Validation
Desktop probe of these files (LiteRT interpreter, XNNPACK CPU, greedy decode; CER vs punctuation-normalized references, whitespace removed). Multilingual clips: Korean sentence + English sentence + short Korean voice commands; .en clips: English sentence pair.
tiny
| Window | Clip | Transcript | CER | Encode s | Decode s (steps) |
|---|
| 5s | 2025년 3월 5일 전술평가 결과 보고 (3.98 s) | 2015년 3월호일 전술 평가 결과 보고 | 0.118 | 0.015 | 0.36 (12) |
| 5s | The current weather in Seoul is cloudy (2.9 s) | The current weather in Seoul is cloudy. | 0.000 | 0.013 | 0.26 (8) |
| 5s | 소리 키워줘 (1.32 s) | 소리 키워줘 | 0.000 | 0.014 | 0.14 (4) |
| 5s | 음량 증가 (1.15 s) | 능량 증가 | 0.250 | 0.015 | 0.23 (6) |
| 10s | 2025년 3월 5일 전술평가 결과 보고 | 2015년 3월호일 전술 평가 결과보고 | 0.118 | 0.046 | 0.43 (13) |
| 10s | The current weather in Seoul is cloudy | The current weather and soil is cloudy. | 0.125 | 0.034 | 0.28 (8) |
| 10s | 소리 키워줘 | 소리 키워줘 | 0.000 | 0.040 | 0.15 (4) |
| 10s | 음량 증가 | 능냥 증가 | 0.500 | 0.040 | 0.24 (6) |
| 30s | 2025년 3월 5일 전술평가 결과 보고 | 2015년 3월호일 전술 평가 결과보고 | 0.118 | 0.214 | 0.67 (13) |
| 30s | The current weather in Seoul is cloudy | The current weather and soul is cloudy. | 0.094 | 0.210 | 0.45 (8) |
| 30s | 소리 키워줘 | 소리 키워줘 | 0.000 | 0.210 | 0.25 (4) |
| 30s | 음량 증가 | 는 양증가 | 0.500 | 0.206 | 0.31 (5) |
The short windows track the checkpoint's own 30 s behavior (same clips right/wrong at every window; residual errors are tiny's model-capacity errors, present at 30 s too) at ~14x encoder speedup for the 5 s window.
tiny.en
| Window | Clip | Transcript | CER | Encode s | Decode s (steps) |
|---|
| 5s | Tactical Evaluation Results Report - March 5, 2025 (4.87 s) | Tactical Evaluation Results Report, March 5, 2025. | 0.000 | 0.016 | 0.29 (10) |
| 5s | The current weather in Seoul is cloudy (2.9 s) | The current weather and soul is cloudy. | 0.094 | 0.014 | 0.28 (8) |
| 10s | Tactical Evaluation Results Report - March 5, 2025 | Tactical Evaluation Results Report, March 5, 2025. | 0.000 | 0.041 | 0.34 (10) |
| 10s | The current weather in Seoul is cloudy | The current weather and soul is cloudy. | 0.094 | 0.038 | 0.28 (8) |
| 30s | Tactical Evaluation Results Report - March 5, 2025 | Tactical Evaluation Results Report. March 5, 2025. | 0.000 | 0.206 | 0.59 (10) |
| 30s | The current weather in Seoul is cloudy | The current weather and soul is cloudy. | 0.094 | 0.202 | 0.44 (8) |
Transcript-identical to the 30 s window on both clips (including the same "and soul" mishearing — a tiny-capacity trait, present at 30 s too) at ~13x encoder speedup for the 5 s window.
base
| Window | Clip | Transcript | CER | Encode s | Decode s (steps) |
|---|
| 5s | 2025년 3월 5일 전술평가 결과 보고 (3.98 s) | 2025년 3월 5일 전술 평가 결과 보고 | 0.000 | 0.039 | 0.62 (12) |
| 5s | The current weather in Seoul is cloudy (2.9 s) | The current weather and soul is cloudy. | 0.094 | 0.045 | 0.44 (8) |
| 5s | 소리 키워줘 (1.32 s) | 소리 키워줘 | 0.000 | 0.031 | 0.23 (4) |
| 5s | 음량 증가 (1.15 s) | 음량 증가 | 0.000 | 0.036 | 0.27 (5) |
| 10s | 2025년 3월 5일 전술평가 결과 보고 | 2025년 3월 5일 전술 평가 결과 보고 | 0.000 | 0.092 | 0.72 (12) |
| 10s | The current weather in Seoul is cloudy | The current weather and soul is cloudy. | 0.094 | 0.080 | 0.53 (8) |
| 10s | 소리 키워줘 | 소리 키워줘 | 0.000 | 0.086 | 0.30 (4) |
| 10s | 음량 증가 | 음량 증가 | 0.000 | 0.090 | 0.33 (5) |
| 30s | 2025년 3월 5일 전술평가 결과 보고 | 2025년 3월 5일 전술 평가 결과 보고 | 0.000 | 0.422 | 1.20 (12) |
| 30s | The current weather in Seoul is cloudy | The current weather in Seoul is cloudy. | 0.000 | 0.428 | 0.85 (8) |
| 30s | 소리 키워줘 | 소리 키워줘 | 0.000 | 0.436 | 0.47 (4) |
| 30s | 음량 증가 | 음량 증가 | 0.000 | 0.449 | 0.60 (5) |
~11x encoder speedup for the 5 s window; base is the recommended minimum for non-English voice commands.
base.en
| Window | Clip | Transcript | CER | Encode s | Decode s (steps) |
|---|
| 5s | Tactical Evaluation Results Report - March 5, 2025 (4.87 s) | Tactical Evaluation Results Report, March 5, 2025. | 0.000 | 0.035 | 0.52 (10) |
| 5s | The current weather in Seoul is cloudy (2.9 s) | The current weather and soul is cloudy. | 0.094 | 0.046 | 0.48 (8) |
| 10s | Tactical Evaluation Results Report - March 5, 2025 | Tactical Evaluation Results Report, March 5, 2025. | 0.000 | 0.080 | 0.62 (10) |
| 10s | The current weather in Seoul is cloudy | The current weather and soul is cloudy. | 0.094 | 0.087 | 0.52 (8) |
| 30s | Tactical Evaluation Results Report - March 5, 2025 | Tactical Evaluation Results Report, March 5, 2025. | 0.000 | 0.449 | 1.05 (10) |
| 30s | The current weather in Seoul is cloudy | The current weather in Seoul is cloudy. | 0.000 | 0.449 | 0.88 (8) |
small
| Window | Clip | Transcript | CER | Encode s | Decode s (steps) |
|---|
| 5s | 2025년 3월 5일 전술평가 결과 보고 (3.98 s) | 2025년 3월 5일 전술평가 결과 보고 | 0.000 | 0.145 | 1.77 (13) |
| 5s | The current weather in Seoul is cloudy (2.9 s) | The current weather in Seoul is cloudy. | 0.000 | 0.145 | 1.17 (8) |
| 5s | 소리 키워줘 (1.32 s) | 소리 키워줘 | 0.000 | 0.155 | 0.64 (4) |
| 5s | 음량 증가 (1.15 s) | 음량 증가 | 0.000 | 0.128 | 0.82 (5) |
| 10s | 2025년 3월 5일 전술평가 결과 보고 | 2025년 3월 5일 전술평가 결과 보고 | 0.000 | 0.311 | 2.27 (13) |
| 10s | The current weather in Seoul is cloudy | The current weather in Seoul is cloudy. | 0.000 | 0.350 | 1.50 (8) |
| 10s | 소리 키워줘 | 소리 키워줘 | 0.000 | 0.339 | 0.83 (4) |
| 10s | 음량 증가 | 음량 증가 | 0.000 | 0.332 | 1.03 (5) |
| 30s | 2025년 3월 5일 전술평가 결과 보고 | 2025년 3월 5일 전술평가 결과 보고 | 0.000 | 1.456 | 4.19 (13) |
| 30s | The current weather in Seoul is cloudy | The current weather in Seoul is cloudy. | 0.000 | 1.446 | 2.73 (8) |
| 30s | 소리 키워줘 | 소리 키워줘. | 0.000 | 1.545 | 1.74 (5) |
| 30s | 음량 증가 | 음량 증가 | 0.000 | 1.495 | 1.85 (5) |
12/12 exact across all three windows — the 5 s window is character-identical to the 30 s window at ~10x encoder speedup.
small.en
| Window | Clip | Transcript | CER | Encode s | Decode s (steps) |
|---|
| 5s | Tactical Evaluation Results Report - March 5, 2025 (4.87 s) | Tactical Evaluation Results Report, March 5, 2025 | 0.000 | 0.138 | 1.30 (9) |
| 5s | The current weather in Seoul is cloudy (2.9 s) | The current weather in Seoul is cloudy. | 0.000 | 0.144 | 1.16 (8) |
| 10s | Tactical Evaluation Results Report - March 5, 2025 | Tactical Evaluation Results Report, March 5, 2025. | 0.000 | 0.312 | 1.74 (10) |
| 10s | The current weather in Seoul is cloudy | The current weather in Seoul is cloudy. | 0.000 | 0.321 | 1.48 (8) |
| 30s | Tactical Evaluation Results Report - March 5, 2025 | Tactical Evaluation Results Report, March 5, 2025. | 0.000 | 1.490 | 3.30 (10) |
| 30s | The current weather in Seoul is cloudy | The current weather in Seoul is cloudy. | 0.000 | 1.516 | 2.56 (8) |
All exact. In the same evaluation pipeline on whisper-base, the ACFT 5 s graph reproduced the stock 30 s transcripts clip-for-clip at ~8x encoder speedup (f32) / ~20x (drq); stock checkpoints at a 5 s window (no ACFT) fall into a repetition loop on sentence-length clips (CER 10.4). Short-window graphs are device-validated on Android arm64 (Snapdragon 865, 0.5–0.8 s end-to-end per command clip for tiny/base 5 s) in the producing project.
Caveats
- The 5 s/10 s graphs hard-truncate longer audio — pick the window that matches your audio length; use the 30 s graph for dictation.
- Decoder is a full-sequence re-run per step (no KV cache) to match the litert-community tiny/base interface — a KV-cached runtime will be substantially faster per token.
- tiny retains its stock model-capacity limits on short noisy/quiet clips; base is the recommended minimum for non-English voice commands.
- Language forcing recommended for short clips on the multilingual models (e.g.
<|ko|> / <|en|>).
Credits and license
- ACFT method and checkpoints: FUTO (
futo-org/acft-whisper-*, Apache-2.0, derived from the MIT openai/whisper-* checkpoints). This repo contains TFLite conversions of those checkpoints, no retraining.
- Conversion and validation: LiteRT-LM-Unity.
- License: Apache-2.0, matching the source ACFT checkpoints.
Related: Korean-fine-tuned ACFT short-window models (tiny/base/medium/large-v3-turbo) live at
leuconoe/whisper-acft-ko.
