Views
No views yet
CohereLabs/cohere-transcribe-03-2026
for faster, lighter CPU inference, plus a small loader so you never have to
load the fp32 weights at runtime.| fp32 (load + quantize each start) | this cached int8 | |
|---|---|---|
| Inference speed | baseline | ~25–30% faster |
| Process RAM | ~14 GB steady / ~22 GB peak | ~6 GB |
| Load time | ~20 s | ~6–11 s |
| Output | — | identical to fp32 |
⚠️ You still need access to the GATED base model. This repo ships only the quantized weights (cohere_int8_state.pt). The config, custom model code and processor are pulled fromCohereLabs/cohere-transcribe-03-2026, which is gated — request access there and be logged in. License/usage follow the base model.
1pip install -r requirements.txt
2# transformers MUST be 5.3.0 — the model's custom code targets 5.3.0.dev0.
3# transformers 5.4.0+ loads the weights cleanly but produces multilingual GARBAGE.1from load_cohere_int8 import load_int8_model
2import soundfile as sf, numpy as np
3
4model, processor = load_int8_model() # downloads cohere_int8_state.pt from this repo
5wav, sr = sf.read("audio_16k_mono.wav") # 16 kHz mono float
6wav = np.asarray(wav if wav.ndim == 1 else wav.mean(1), dtype=np.float32)
7
8out = model.transcribe(processor=processor, language="en",
9 audio_arrays=[wav], sample_rates=[sr], punctuation=True)
10print(out[0])language is required (no auto-detect). Supported: en, fr, de, es, it, pt, nl, pl,
el, ar, ja, zh, vi, ko.1from transformers import AutoModelForSpeechSeq2Seq
2import torch
3m = AutoModelForSpeechSeq2Seq.from_pretrained("CohereLabs/cohere-transcribe-03-2026",
4 dtype=torch.float32, low_cpu_mem_usage=True, trust_remote_code=True).eval()
5mq = torch.ao.quantization.quantize_dynamic(m, {torch.nn.Linear}, dtype=torch.qint8)
6torch.save(mq.state_dict(), "cohere_int8_state.pt") # ~2.7 GBload_cohere_int8.py) then rebuilds the architecture on the meta
device (zero weight memory), swaps each nn.Linear for an empty dynamic-int8
Linear, and load_state_dict(..., assign=True) — so fp32 is never allocated.CohereLabs/cohere-transcribe-03-2026. This is an
unofficial quantized redistribution of the weights for convenience; all rights,
license terms and gating of the base model apply.