ArSL-Models — Arabic Sign Language Recognition

⚠️ Note: Quantitative results below (accuracy, F1, dataset sizes) are placeholders pending the accompanying paper. They will be updated with the exact reported figures.

---

1. Alphabet model — improved_arsl_model.pth

• Backbone: ResNet18 (ImageNet-pretrained), final pooling/fc removed → 512×7×7 feature map.
• Sequence: the 7×7 grid is flattened to a length-49 sequence of 512-d vectors.
• Recurrence: 2-layer bidirectional LSTM, hidden size 512 (→ 1024-d outputs).
• Attention: additive attention pools the LSTM outputs into one context vector.
• Head: 1024 → 512 → 256 → 29 MLP with BatchNorm, ReLU, dropout (0.5).

• RGB image, resized to 224×224, normalized with ImageNet mean/std ([0.485, 0.456, 0.406] / [0.229, 0.224, 0.225]).
• Hand presence is verified with MediaPipe before classification.

Alphabet label map (index → letter)

Reported metrics (to update from paper)

---

2. Word model — sign_word_t5_classifier_best_3d.pth

• Base: encoder of google-t5/t5-small (d_model = 512).
• Input projection: Linear(feature_dim → 512) → Dropout → LayerNorm → GELU.
• Pooling: first-token (CLS-style) hidden state of the encoder.
• Head: Dropout → Linear(512 → 10).

• MediaPipe hand landmarks: 21 landmarks × 3 coords (x, y, z) for 1 hand → feature_dim = 63.
• Landmarks are wrist-centered and scaled by the wrist→middle-MCP distance.
• A single frame's landmarks are tiled to a sequence length of 100 with an all-ones attention mask.

Word label map (index → Arabic → English)

Reported metrics (to update from paper)

---

Usage

Intended use & limitations

• Intended use: education, accessibility demos, and research on Arabic sign language recognition.
• Limitations: trained on a limited label set (29 letters / 10 words); accuracy depends on lighting, camera angle, hand visibility, and signing style. The word model classifies from a single tiled frame and is not a full continuous-sign sequence model. Not validated for clinical or safety-critical use.

Citation

License