Architecture: End-to-end (E2E) visual-only speech recognition, 250M parameters
Modality: Video only (mouth ROI crops, no audio)
Frontend: MediaPipe face detection + mouth ROI cropping at 16-second segments
Training: 20 epochs on a single NVIDIA A100-SXM4-80GB, learning rate 0.001
Framework: PyTorch 2.5.1 + PyTorch Lightning 2.5.0
Checkpoint: model_avg_10.pth — average of the last 10 epoch checkpoints

Dataset

Trained on the supervised trainval split from Ainncy/LRS3:

Training samples: ~98% of trainval (lrs3_train_fit.csv)
Validation samples: ~2% of trainval (lrs3_train_val.csv)
Note: The official LRS3 test set is not distributed by Ainncy/LRS3. Validation metrics are computed on a deterministic held-out subset of trainval and should not be compared to published LRS3 test-set results.

Intended Use

This model is intended for research purposes only in visual speech recognition (lip reading). It does not use audio input and is designed to explore the limits of visual-only speech recognition.

Limitations

Visual-only: Performance is inherently lower than audio-visual or audio-only models.
English only: Trained solely on English speech from LRS3 (TED/TEDx talks).
Controlled environment: LRS3 consists of frontal-facing speakers with relatively stable lighting. Real-world performance will be significantly lower.
No speaker diarization: Cannot distinguish between multiple speakers.
Vocabulary constrained: Limited to vocabulary present in the training data.
Privacy & ethics: Lip-reading technology raises privacy concerns. This model should not be used for surveillance or non-consensual speech recognition.

Training Setup

Preprocessing was performed on Modal cloud GPUs using the pipeline in the accompanying repository:

Mouth ROI detection with MediaPipe (CPU, 8 parallel shards)
Video cropping to 16-second segments
Training on 1× A100-80GB for 20 epochs (~4h40)

The raw MP4 video files, crops, and transcriptions are not included in this repository and must not be redistributed without explicit authorization from the dataset copyright holders.

Citation

bibtex

@article{ma2023auto,
  title={Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels},
  author={Ma, Pingchuan and Haliassos, Alexandros and Fernandez-Lopez, Adriana
          and Chen, Honglie and Petridis, Stavros and Pantic, Maja},
  journal={arXiv preprint arXiv:2303.08807},
  year={2023}
}

License

This model is released under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0), consistent with the LRS3 dataset terms. Commercial use is prohibited.

simonlesaumon/lrs3-lipreader-visual-only

LRS3 Visual-Only Lip Reading Model

Model Description