Saudi Sign Language — Continuous Sign Language Recognition (CSLR)

What this builds

Project layout

saudi_cslr/
├── configs/
│   └── default.yaml              # all hyperparameters
├── data_module/
│   ├── vocabulary.py             # gloss <-> id mapping (idx 0 = <blank>)
│   ├── transforms.py             # frame-level + clip-level augmentations
│   └── dataset.py                # frame-folder Dataset + collate_fn
├── models/
│   ├── visual_encoder.py         # Swin-T / ResNet backbones
│   ├── temporal_module.py        # MSTP stack
│   ├── sequence_model.py         # packed BiLSTM
│   └── cslr_model.py             # the full model with two heads
├── losses/
│   └── cslr_loss.py              # CTC + VAC + SMKD
├── utils/
│   ├── decoder.py                # greedy + beam-search CTC decoders
│   ├── metrics.py                # WER (Levenshtein-based)
│   └── train_utils.py            # AverageMeter, warmup scheduler, checkpointing
├── train.py                      # main training entry point
├── eval.py                       # standalone evaluation script
├── requirements.txt
└── README.md

Expected data format

videos/
├── 00_0001/
│   ├── frame_001.jpg
│   ├── frame_002.jpg
│   └── ...
├── 00_0002/
└── ...

annotations/
├── train.txt
├── dev.txt
└── test.txt

id|gloss|text
00_0001|سوال هو|من هو
00_0002|هو معلم لغه اشاره|هو مدرس لغه اشاره
00_0003|استفهام هو معلم هو|هل انت مدرس
00_0004|هو معلم لا انا مدرسه|انت لست مدرس انا طالب

• id — must match the frame folder name under videos/.
• gloss — space-separated SSL gloss tokens. This is the training target for CSLR.
• text — Arabic spoken-language translation. Carried through the pipeline but not used by the CSLR loss (it's reserved for the SLT extension).

Setup

Train

1. Build a gloss vocabulary from train.txt (saved to work_dir/vocab.json).
2. Train for train.epochs epochs with:
   • Mixed precision (FP16)
   • Gradient accumulation (effective batch = batch_size * grad_accum_steps)
   • Linear warmup followed by multistep LR decay
   • Different LR for the backbone (lr * backbone_lr_mult) and the head
3. Evaluate on the dev set after every epoch using beam-search CTC decoding and report WER (the standard CSLR metric).
4. Save:
   • work_dir/checkpoints/best.pth — best dev-WER checkpoint
   • work_dir/checkpoints/epoch_NNN.pth — last few epoch checkpoints
   • work_dir/train.log — full training log

Evaluate

How the loss works

                                  ┌──── visual_classifier ──→ vis_logits ──┐
                                  │                                          │   CTC(vis)  ← VAC
frames → backbone → MSTP ─────────┤                                          ├── SMKD(KL bidirectional)
                                  │                                          │
                                  └──── BiLSTM ── seq_classifier ─→ seq_logits ──→ CTC(seq) ← main

L_total =  ctc_weight · CTC(seq_logits, labels)        # main supervision
        + vac_weight · CTC(vis_logits, labels)         # auxiliary supervision (VAC)
        + smkd_weight · T² · 0.5 · [ KL(seq‖vis) + KL(vis‖seq) ]   # mutual KD (SMKD)

Tuning notes (in order of impact)

1. model.backbone — swin_tiny is best; switch to resnet18 only for tight VRAM.
2. loss.smkd_weight — 25 is the sweet spot per the paper; try 15-50 if you have compute to grid-search.
3. train.augmentations.color_jitter_strength — 0.4 works well for the "in-the-wild" smartphone footage that Isharah uses; lower it for cleaner studio data.
4. data.max_frames — the paper has clips up to ~700 frames. The default 300 uniformly subsamples longer clips. Increase if you have VRAM headroom.
5. optim.backbone_lr_mult — keep at 0.1 when starting from ImageNet weights; raise to 0.5 once fine-tuned for a few epochs.

Expected results

Caveats

• Horizontal flipping is disabled by default — sign language is direction- sensitive, so flipping changes meaning. Don't enable it.
• The gloss vocabulary is built once from the training file and cached in work_dir/vocab.json. Delete that file if you change train.txt or the vocabulary will be stale.
• The provided beam decoder is pure-Python and adequate for ~1k-sample dev sets. For full Isharah-2000 test runs (8 000 samples), consider plugging in a C++ CTC beam decoder (e.g. pyctcdecode) — the architecture is unchanged.