It ships compiled .mlmodelc bundles, ready for on-device inference through WhisperKit's Swift API. For Python inference, use the upstream repo linked above.

Output script: Devanagari for Hindi, Latin for English, switching mid-sentence as the speaker does.

Usage

swift

import WhisperKit

let pipe = try await WhisperKit(
    WhisperKitConfig(
        model: "Trelis_whisper-hinglish-preview",
        modelRepo: "vbhar/whisperkit-hinglish-large-v3-coreml"
    )
)
let result = try await pipe.transcribe(audioPath: "meeting.wav")

Or download the variant folder directly and point WhisperKit at a local path:

bash

hf download vbhar/whisperkit-hinglish-large-v3-coreml \
  --include "Trelis_whisper-hinglish-preview/*" \
  --local-dir ./models

Layout

Trelis_whisper-hinglish-preview/
├── MelSpectrogram.mlmodelc/
├── AudioEncoder.mlmodelc/     # ~1.2 GB weights
├── TextDecoder.mlmodelc/      # ~1.7 GB weights
├── config.json
└── generation_config.json

No quantized variants and no decoder prefill data are included — float16 only.

Attribution and modifications

Lineage:


This repo	`vbhar/whisperkit-hinglish-large-v3-coreml` — Core ML float16 conversion
Direct source	`Trelis/whisper-hinglish-preview` — Apache-2.0
Its base	`ARTPARK-IISc/whisper-large-v3-vaani-hindi`
Upstream architecture	`openai/whisper-large-v3`

All credit for the model itself belongs to Trelis and, upstream of them, to ARTPARK-IISc and OpenAI. Licensed under the Apache License, Version 2.0; see https://www.apache.org/licenses/LICENSE-2.0.

Statement of modifications (Apache-2.0 §4(b)): the weights were converted from float32 to float16 and compiled to Core ML for the Apple Neural Engine. They are otherwise unmodified — no fine-tuning, no distillation, no pruning, no re-training. One structural change was required to make the model usable at all: the output embedding was resized from 51867 to 51866 tokens, dropping the trailing <|mixedcode|> token row. See below for why this is necessary and what it costs.

Reproducing this conversion

Converted with argmaxinc/whisperkittools under Python 3.11 (3.14 has no working coremltools/torch wheels as of this writing):

bash

python3.11 -m venv .venv && source .venv/bin/activate
pip install git+https://github.com/argmaxinc/whisperkittools.git

whisperkit-generate-model \
  --model-version Trelis/whisper-hinglish-preview \
  --output-dir out \
  --repo-path-suffix fp16

float16, no quantized variants, no decoder prefill data. Apply the vocab fix below before the decoder is traced.

Conversion gotcha: the 51867th token silently breaks the tokenizer

Trelis/whisper-hinglish-preview has vocab_size: 51867 — one token more than whisper-large-v3's 51866. The extra entry is a <|mixedcode|> control token.

That single extra row is enough to produce a model that loads, runs, and emits garbage, with no error anywhere:

WhisperKit selects its tokenizer solely from the width of the decoder's logits tensor.
51866 → it loads the openai/whisper-large-v3 tokenizer.
Anything else → it silently falls back to openai/whisper-base.
whisper-base is a 51865-token vocabulary with different special-token ids. Every <|startoftranscript|>, language, task and timestamp id is shifted, so decoding produces plausible-looking but wrong text — or control-token soup. Nothing throws.

Fix: shrink the vocabulary back to 51866 before conversion.

python

from transformers import WhisperForConditionalGeneration

model = WhisperForConditionalGeneration.from_pretrained("Trelis/whisper-hinglish-preview")
model.resize_token_embeddings(51866)   # drops the trailing <|mixedcode|> row

proj_out is weight-tied to the input embedding, so this one call resizes both the embedding matrix and the decoder's output projection; the resulting logits tensor is [1, 1, 51866] and WhisperKit picks the correct large-v3 tokenizer. The cost is that <|mixedcode|> becomes unaddressable — in practice it is not needed, as the model code-switches from context anyway.

The converted decoder in this repo is verified to emit logits: [1, 1, 51866], so consumers of this repo do not need to do anything — the fix matters only if you re-convert from the Trelis source yourself.

A note on quantization

Float16 was chosen deliberately. During this work, a quantized Core ML build of these weights was observed to emit the literal string nan on real code-switched Hindi/English speech, while passing cleanly on short English clips such as WhisperKit's bundled jfk.wav.

The asymmetry is the point worth noting: quantization error is input-dependent, so a short smoke test — the normal way to sanity-check a conversion — can pass on a build that fails on the audio you actually care about. If you evaluate a quantized Whisper conversion, test it on your own code-switched audio at realistic length, not on a short clip.

No quantized variant is published here, because none was validated on that kind of input.

Limitations

~2.9 GB on disk. Full large-v3 parameter count at float16.
First load is slow. The Neural Engine compiles the model once on first use: roughly 115–190 s on an M-series Mac. Subsequent warm loads are ~1–3 s. Cache the compiled artifacts and show progress; do not block a user on the cold path silently.
Apple Silicon only. Requires macOS 14+ / iOS 17+ and an ANE.
<|mixedcode|> is gone by construction (see above). Code-switching still works; the token just cannot be forced.
Script behaviour is not configurable. This model writes Hindi in Devanagari and English in Latin. If you want Hindi romanized into Latin script, this is the wrong model — use vbhar/whisperkit-hindi2hinglish-prime-coreml instead. Picking the wrong one of the two is the most likely mistake with this pair.
Marked preview upstream; inherits the biases and domain limits of the Vaani-Hindi base.

Verified usage

Tested live end-to-end in a real macOS menu-bar meeting-notes app (mic + system audio, streaming transcription) on natural Hindi/English code-switched speech.

Hindi is returned in Devanagari and English in Latin script, mixed inline within the same sentence.