A streaming speech-to-text model fine-tuned from
kyutai/stt-1b-en_fr for Icelandic.
The fine-tune extends the text vocabulary with Icelandic sub-words and adds two task-domain prompts so the same checkpoint can either
transcribe Icelandic or
translate Icelandic → English.
The model was trained with a 5 second delay, so when using it, be sure to append a 5-second delay to your audio input.
1import torch
2from transformers import (
3 KyutaiSpeechToTextForConditionalGeneration,
4 KyutaiSpeechToTextProcessor,
5)
6from transformers.generation import LogitsProcessor
7
8DOMAIN_TOKENS = {
9 "asr_is_is": [9318, 8002, 8003, 9193], # <is> — Icelandic ASR
10 "asr_is_en": [9318, 8032, 8015, 9193], # <en> — Icelandic → English
11}
12
13
14class ForcePrefix(LogitsProcessor):
15 """Force the first N tokens to the domain prefix, then force <pad> for
16 `pad_steps` more steps (the model's asr_delay window). Without the pad
17 window the model can prematurely emit an end-of-utterance token (\\n / .)
18 and stay in pad mode forever, producing no transcript.
19 """
20 def __init__(self, prefix, prompt_len=1, pad_steps=12, pad_token_id=3):
21 self.prefix = prefix
22 self.prompt_len = prompt_len
23 self.pad_steps = pad_steps
24 self.pad_token_id = pad_token_id
25
26 def __call__(self, input_ids, scores):
27 i = input_ids.shape[1] - self.prompt_len
28 if 0 <= i < len(self.prefix):
29 scores[:] = float("-inf")
30 scores[:, self.prefix[i]] = 0.0
31 elif i < len(self.prefix) + self.pad_steps:
32 scores[:] = float("-inf")
33 scores[:, self.pad_token_id] = 0.0
34 return scores
35
36
37device = (
38 "cuda" if torch.cuda.is_available()
39 else "mps" if torch.backends.mps.is_available()
40 else "cpu"
41)
42
43model_id = "mideind/kyutai-stt-1b-is-en"
44processor = KyutaiSpeechToTextProcessor.from_pretrained(model_id)
45model = KyutaiSpeechToTextForConditionalGeneration.from_pretrained(
46 model_id, torch_dtype=torch.bfloat16
47).to(device).eval()
48
49# audio: 1-D float32 numpy array, 24 kHz mono, [-1, 1] range.
50# IMPORTANT: append ~5 s of trailing silence so the model has time to flush
51# its delayed text output past the asr_delay boundary.
52import numpy as np
53audio_with_silence = np.concatenate([audio, np.zeros(5 * 24000, dtype=np.float32)])
54
55inputs = processor(audio=audio_with_silence, sampling_rate=24000)
56inputs = {k: v.to(device) for k, v in inputs.items()}
57
58domain = "asr_is_is" # or "asr_is_en"
59with torch.no_grad():
60 out = model.generate(
61 **inputs,
62 logits_processor=[ForcePrefix(DOMAIN_TOKENS[domain])],
63 )
64
65text = processor.batch_decode(out, skip_special_tokens=True)[0]
66print(text)
The fine-tune supports two task-domain prompts. They must be injected as the first 4 generated text tokens (see ForcePrefix above). Without one of them, the model produces only padding tokens.
Same architecture as
kyutai/stt-1b-en_fr-trfs with an extended text vocabulary. Pure decoder-only transformer over text + audio codebooks, with a built-in Mimi audio codec (inlined into the same checkpoint).
Evaluated on 100 examples from a filtered subset of the Samrómur test split. Metrics are reported both raw (case- and punctuation-sensitive) and normalised (lowercased, punctuation stripped, whitespace squeezed).
The translation BLEU looks very good on this in-domain sample, but the model does not generalise well outside the samrómur read-speech distribution — expect significant quality degradation on conversational, accented, noisy, or otherwise out-of-distribution audio.
