MSP-VSR is the visual speech-recognition model in
Multimodal Speech Perception (MSP). It uses an AV-HuBERT encoder and a CTC head to transcribe English speech from silent mouth-region video.
The processor converts frames to grayscale, rescales and normalizes them, resizes to 96 pixels, and applies an 88-pixel crop. MSP-VSR also supplies the visual encoder to MSP-AVSR, where its representations participate in both directions of the bidirectional cross-attention fusion block.
1import torch
2from transformers import AutoModelForCTC, AutoProcessor
3
4model_id = "MahmoodAnaam/MSP-VSR"
5processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
6model = AutoModelForCTC.from_pretrained(model_id, trust_remote_code=True).eval()
7
8inputs = processor(videos="sample.mp4", return_tensors="pt")
9with torch.inference_mode():
10 logits = model(**inputs).logits
11
12text = processor.tokenizer.batch_decode(logits.argmax(dim=-1))[0]
13print(text)
Use footage with a visible, trackable speaking face. For 3-gram beam search, install pyctcdecode and kenlm, then decode with MahmoodAnaam/MSP-Processor-With-LM. Pin Hub revisions when loading custom code in controlled environments.
