Sophea-Canary-ASR

Name: Sophea-Canary-ASR
Availability: InStock

A bilingual (Greek + English), multitask speech model fine-tuned from nvidia/canary-1b-v2 (1 B, offline attention-encoder-decoder, FastConformer).

Note on Greek final sigma (inherited from `canary-1b-v2`)

The base nvidia/canary-1b-v2 SentencePiece tokenizer has no token for the Greek word-final sigma ς (it folds to <unk>), together with a handful of other Greek characters — this gap ships with the base canary model itself, not with this fine-tune. An earlier version of this checkpoint inherited that gap and emitted ⁇ in place of every word-final ς. The current weights fix it: the model was retrained on tokenizer-normalized Greek targets, so it now outputs the medial form σ everywhere and never <unk>. When serving, restore display orthography with a deterministic word-final σ -> ς rule in post-processing.

Base model vs. this model

How much did fine-tuning move each task? Both the base nvidia/canary-1b-v2 and this model were scored on the same held-out test manifests with the same validation-path harness and literal WER (casing + punctuation), so these deltas are directly comparable.

Task	Test set	Base `canary-1b-v2`	This model	Δ
Greek ASR	FLEURS (el)	30.93 % WER	12.69 % WER	−18.2
Greek ASR	Common Voice 17 (el)	37.30 % WER	2.90 % WER	−34.4
English ASR	FLEURS (en)	12.02 % WER	12.90 % WER	+0.9 (retained)
Greek->English	FLEURS (el->en)	BLEU 25.46	BLEU 24.13	−1.3 (retained)

Reading it: Greek fine-tuning is the whole story — FLEURS WER drops by more than half (30.9 -> 12.7) and Common Voice collapses from 37.3 to 2.9. English ASR and Greek->English translation were never trained here; they only moved by ~1 point, i.e. they were retained rather than forgotten. Canary's source_lang/target_lang prompt-conditioning is what keeps the English and translation pathways intact while the Greek pathway is rebuilt.

What it does

One model does four jobs:

Task	Test set	Score
Greek ASR	FLEURS (el)	12.69 % WER
Greek ASR	Common Voice 17 (el)	2.90 % WER
English ASR	FLEURS (en)	12.90 % WER
Greek -> English speech translation	FLEURS (el->en)	BLEU 24.13

It also handles medical and legal Greek domains (in the fine-tuning mix). This is the offline, high-accuracy model of a two-model Greek ASR system; a separate streaming 0.6 B model serves the real-time path.

Highlights

Halves Greek WER vs the streaming baseline (25.5 % -> 12.7 % FLEURS) and drives Common Voice to 2.9 %.
Bilingual for free. Despite fine-tuning on ~360 h of mostly-Greek data, English ASR held at 12.9 % (base canary-1b-v2: 12.0 %) — Canary's source_lang/target_lang prompt-conditioning preserves the English pathway (no catastrophic forgetting).
Translation for free. Greek->English speech translation (BLEU 24.1) was retained from the base through ASR-only fine-tuning — the prompt selects the task, so the AST pathway survives.

Usage

Requires NeMo (pip install nemo_toolkit[asr]).

python

from nemo.collections.asr.models import EncDecMultiTaskModel
from huggingface_hub import hf_hub_download

ckpt = hf_hub_download("KIEFERSA/Sophea-Canary-ASR",
                       "canary1bv2_el_stage3.nemo")
model = EncDecMultiTaskModel.restore_from(ckpt)

# Greek ASR
print(model.transcribe(["greek_audio.wav"], source_lang="el", target_lang="el", pnc="yes"))

# English ASR
print(model.transcribe(["english_audio.wav"], source_lang="en", target_lang="en", pnc="yes"))

# Greek -> English speech translation
print(model.transcribe(["greek_audio.wav"], source_lang="el", target_lang="en", pnc="yes"))

Prompt-sensitivity note. This model is fine-tuned with the canary2 prompt and pnc=yes. Pass source_lang/target_lang explicitly (as above) and keep pnc=yes for results matching the reported metrics. Audio should be 16 kHz mono.

Training

Base: nvidia/canary-1b-v2 (1 B, offline AED, FastConformer encoder).
Method: full fine-tune via a custom NeMo launcher that restores the pretrained model and swaps the dataset (avoiding the tokenizer rebuild ..), with a 3-stage warm-start chain (each stage initialized from the previous endpoint):
1. stage-1 — ~250 h Greek ASR (FLEURS + Common Voice + YODAS + TEDx + targeted TTS).
2. stage-2 — + ~80 h clean Greek (medical, legal, parliamentary) -> 330 h.
3. stage-3 (this model) — + 30 h English ASR (LibriSpeech-clean) -> 360 h.
Optimizer: AdamW, cosine schedule, lr ~8e-6 (stage-3), bf16, Lhotse dynamic batching.
Checkpoint selection: best Greek val_wer (0.0577 at stage-3).

Data

Source	Role	Notes
FLEURS (el / en)	Greek + English ASR	clean read speech
Common Voice 17 (el)	Greek ASR	spontaneous, multi-speaker
YODAS (el)	Greek ASR	YouTube subtitles, quality-filtered
TEDx (el)	Greek ASR	real talks
TTS-synthetic (el)	Greek ASR	Wikipedia/general domain, ~10 % of mix
Medical / legal (el)	Greek ASR	domain coverage
LibriSpeech (en)	English ASR	retention

Limitations

Offline only. Full-context AED — not for streaming/real-time use.
Translation ceiling. Greek->English BLEU (~24) is inherited from the base; an explicit AST fine-tune on machine-translated data did not improve it (you cannot out-train your labels). Real parallel data would be needed to push past this.
Domain bias. Strongest on read speech (FLEURS) and Common Voice; other domains (e.g. heavy dialect, far-field, overlapping speech) are untested.
Common Voice over-specialization. The validation mix is CV-heavy, so CV WER (2.9 %) is partly in-distribution; FLEURS (12.7 %) is the more conservative real-world estimate.

License

Released under CC-BY-4.0, inheriting the license of the base model nvidia/canary-1b-v2.

Citation

If you use this model, please credit this repository and the base model:

bibtex

@misc{kiefer2026canarygreek,
  title  = {Sophea-Canary-ASR: a bilingual Greek+English multitask speech model},
  author = {Kirouane, Ayoub},
  year   = {2026},
  howpublished = {Hugging Face, KIEFERSA},
  note   = {Fine-tuned from nvidia/canary-1b-v2}
}

KIEFERSA/Sophea-Canary-ASR