How2Sign Phoneme Prediction (Auto-AVSR)
Visual-only phoneme prediction model fine-tuned from Auto-AVSR on How2Sign lip crops.
Architecture
- Frontend: 3D Conv + ResNet-18 (frozen, pretrained on LRS3+VoxCeleb2)
- Encoder: 12-layer Conformer (first 6 frozen, last 6 fine-tuned)
- CTC Head: Linear(768, num_phonemes) trained from scratch
Input
- Grayscale lip crop videos:
B x 1 x T x 96 x 96
- Normalized to [0, 1]
Output
- CTC phoneme logits, decoded via greedy or beam search
Training Data
- bdanko/how2sign-lip-crops-96x96x1
- Phoneme targets generated with
phonemizer (espeak-ng, en-us)
Usage
See the training notebook for full details.
