Custom HF Inference Endpoint handler that serves whisper-large-v3 via faster-whisper / CTranslate2 instead of the HF transformers pipeline. It is a protocol drop-in for ../endpoint-whisper: same POST {"inputs": <base64 audio>} → {"text": ...}, so src/transcribers/whisper.py (device: api) needs no change — only the endpoint URL differs.

endpoint-whisper/ serves both whisper-large-v3 and the monolingual WAXAL whisper-small-waxal-* fine-tunes, which are in HF transformers format. CTranslate2 needs its own converted format, so we can't swap the shared handler without breaking the WAXAL endpoints. zh runs on its own endpoint (per-language WHISPER_LANG, see whisper.py:_api_url_for), so it gets its own CT2 handler and the WAXAL endpoints stay on the transformers handler + its tokenizer patch.

CTranslate2 is typically ~3–5× faster than the transformers pipeline on the same GPU, at no extra hourly cost — a bigger win than moving A10G→A100 (which buys ~2× for 3–4× the cost). It also cuts VRAM enough that a T4 comfortably serves it (the deploy default).

The four segment-drop knobs above target the residual dropped-span losses seen with VAD already off (faster-whisper discards segments that trip its no-speech/log-prob/ compression thresholds). deploy_whisper_ct2_endpoint.py --drop-fix sets the recommended combo (no_speech+log_prob disabled, condition_on_previous_text off) in one recreate. | ASR_MAX_DECODED_SAMPLES / ASR_MAX_ENCODED_BYTES | — | request bounds (_audio_io.py), raise for 2h shows |

_audio_io.py is a byte-identical copy of the other endpoint*/ copies (CI hash-diff guards it) — keep it in sync; don't edit only this one.

Then paste the URL into channels.taiwan.yaml whisper.api_urls.zh (replacing the transformers endpoint) once status=running. Roll back by pasting the old URL — nothing else changes.

CTranslate2 GPU needs CUDA 12 + cuDNN 9 shared libs in the image. The HF default PyTorch image usually has them; if cold start errors with a missing libcudnn* / libcublas*, add the pip cuDNN wheels to requirements.txt:

nvidia-cudnn-cu12
nvidia-cublas-cu12

(faster-whisper's install docs cover the LD_LIBRARY_PATH variant if the wheels alone don't resolve it.)

float16/int8 quantization can nudge accuracy. Re-run the zh blind judge through the eval harness against the current transformers endpoint on the same Traditional- Chinese set that shipped zh→whisper:

Ship only if CT2 holds parity (it should be faster at equal quality).