A per-language CTC adapter for the MMS-1B model. The ~1B-parameter base model is frozen; only the language adapter (adapter_attn_dim=16) and a fresh Latin-script CTC head are trained. Input is 16 kHz mono audio (up to 30 seconds). Output is a Romanized (Latin-script) transcript, lower-cased and NFC-normalized.

Base model: facebook/mms-1b-all (frozen)
Adapter dimension: adapter_attn_dim = 16
Decoding: greedy CTC, no language model

Datasets

Training / evaluation: sulabhkatiyar/ne-asr-dataset-clk
Augmented training: sulabhkatiyar/ne-asr-dataset-clk-aug

Data source & attribution. The training and evaluation data is derived from the ARTPARK-IISc Vaani project (https://vaani.iisc.ac.in/), released under CC-BY-4.0. Please retain that attribution when you use these models.

Evaluation

Held-out test split, scored per-sample with greedy CTC decoding (batch size 1, no language model). Reference processing: NFC + strip + lower. These are the honest per-sample numbers; they supersede any earlier aggregate figures.

split	n (utterances)	WER %	CER %
test	89	62.88	17.95

How to load

Part of the NE Speech AI V1 collection.

This repo hosts ONLY the per-language adapter and the CTC head — NOT the frozen ~1B facebook/mms-1b-all base weights. As a result:

Wav2Vec2ForCTC.from_pretrained("sulabhkatiyar/ne-asr-clk") does NOT work — the repo hosts only the per-language adapter + CTC head, not the frozen ~1B base; the base must be pulled from facebook/mms-1b-all.
A naive model.load_adapter(...) onto stock facebook/mms-1b-all shape-crashes for the dim-64 languages (lus/grt) — the base must first be built at the right adapter_attn_dim. The snippet below handles this automatically by reading the adapter.clk.meta.json sidecar.

The snippet below is fully self-contained — it needs only public packages and this repo (plus facebook/mms-1b-all for the frozen base), no private code. It reads the adapter dim from the meta sidecar, rebuilds mms-1b-all at that dim, loads the adapter + CTC head, and runs greedy CTC decoding.

python

# pip install "transformers>=5" huggingface_hub safetensors torch
from transformers import (Wav2Vec2ForCTC, Wav2Vec2Config,
                          Wav2Vec2CTCTokenizer, Wav2Vec2Processor,
                          AutoFeatureExtractor)
from huggingface_hub import hf_hub_download
from huggingface_hub.utils import EntryNotFoundError
from safetensors.torch import load_file
import json, torch

BASE = "facebook/mms-1b-all"

def load_ne_asr(repo_id, iso):
    # Reads the per-language adapter dim from the repo's meta sidecar
    # (64 for lus/grt, 16 otherwise; falls back to 16 if absent).
    try:
        meta = json.load(open(hf_hub_download(repo_id, f"adapter.{iso}.meta.json")))
        dim  = int(meta.get("adapter_attn_dim", 16))
    except EntryNotFoundError:
        dim  = 16
    tok  = Wav2Vec2CTCTokenizer.from_pretrained(repo_id)
    feat = AutoFeatureExtractor.from_pretrained(BASE)          # MMS's own feature extractor
    processor = Wav2Vec2Processor(feature_extractor=feat, tokenizer=tok)

    cfg = Wav2Vec2Config.from_pretrained(BASE)
    cfg.adapter_attn_dim  = dim
    cfg.vocab_size        = len(tok)
    cfg.pad_token_id      = tok.pad_token_id
    cfg.ctc_zero_infinity = True

    model = Wav2Vec2ForCTC.from_pretrained(BASE, config=cfg,   # frozen ~1B base (~3.6GB, once)
                                           ignore_mismatched_sizes=True)
    model.init_adapter_layers()
    state = load_file(hf_hub_download(repo_id, f"adapter.{iso}.safetensors"))
    model.load_state_dict(state, strict=False)                # base from mms-1b-all; adapter+head from repo
    return model.eval(), processor

# --- usage (replace with this repo's id + iso) ---
model, processor = load_ne_asr("sulabhkatiyar/ne-asr-clk", "clk")

import torchaudio
wav, sr = torchaudio.load("example.wav")
if sr != 16000:
    wav = torchaudio.functional.resample(wav, sr, 16000)
inputs = processor(wav.squeeze().numpy(), sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    logits = model(inputs.input_values).logits
pred = processor.batch_decode(logits.argmax(-1))[0]
print(pred)

Limitations

The base facebook/mms-1b-all encoder (~1B parameters) is frozen; only a small per-language adapter (adapter_attn_dim=16) and a fresh Latin-script CTC head are trained. Capacity for this language is therefore limited.
Idu Mishmi is a low-resource language; the training data is small and may not cover the full range of speakers, dialects, domains, and recording conditions.
Evaluation used 89 held-out test utterances (per-sample greedy CTC, no language model).

Citation

If you use this model, please cite the MMS base model:

bibtex

@article{pratap2023scaling,
  title   = {Scaling Speech Technology to 1,000+ Languages},
  author  = {Pratap, Vineel and Tjandra, Andros and Shi, Bowen and Tomasello, Paden and Babu, Arun and Kundu, Sayani and Elkahky, Ali and Ni, Zhaoheng and Vyas, Apoorv and Fazel-Zarandi, Maryam and Baevski, Alexei and Adi, Yossi and Zhang, Xiaohui and Hsu, Wei-Ning and Conneau, Alexis and Auli, Michael},
  journal = {arXiv preprint arXiv:2305.13516},
  year    = {2023}
}

And the NE-ASR adapter release (placeholder; replace when the canonical publication is available):

bibtex

@misc{katiyar2026neasr,
  author       = {Katiyar, Sulabh},
  title        = {NE-ASR: MMS-1B Adapters for Northeast Indian Languages},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/sulabhkatiyar/ne-asr-clk}},
  note         = {Placeholder citation; replace with the canonical publication when available.}
}

License

CC-BY-NC-4.0. This adapter is derived from facebook/mms-1b-all, released under CC-BY-NC 4.0. When you use this adapter you must comply with the MMS license terms. The underlying training and evaluation data is derived from the ARTPARK-IISc Vaani project (https://vaani.iisc.ac.in/), released under CC-BY-4.0; please also retain that attribution.