ASL Citizen processed keypoints-200
        ↓
Small Transformer isolated sign recognition model
        ↓
Comparison with BiGRU + Attention baseline
        ↓
Select stronger encoder for future How2Sign CTC stage

Dataset

Training dataset:

SharoonArshad/asl-citizen-processed-200

The dataset uses the processed ASL Citizen keypoints-200 format:

(N, 200, 450)

Where:

text

= fixed sequence length
= 225 position features + 225 velocity features
= 75 landmarks × 3 coordinates
landmarks = 33 pose + 21 left hand + 21 right hand

Model Architecture

The model uses a small Transformer encoder over temporal keypoint sequences:

text

Input:                  (B, 200, 450)
Input LayerNorm:        450
Linear Projection:      450 → 256
CLS Token:              prepended to the sequence
Positional Encoding:    sinusoidal
Transformer Encoder:    3 layers
Attention Heads:        4
Feedforward Dimension:  768
Pooling:                CLS token
Classifier Head:        200 ASL classes

Training Configuration

Main training settings:

text

Batch size: 32
Max epochs: 60
Best epoch: 52
Early stopping patience: 10
Learning rate: 0.0002
Weight decay: 0.0001
Dropout: 0.3
Label smoothing: 0.1
Gradient clipping: 1.0
Scheduler: cosine
Optimizer: AdamW
Mixed precision: enabled on GPU

Results

Small Transformer final results:

text

Best epoch: 52
Best validation Top-1 accuracy: 78.13%
Test loss: 1.3240
Test correct: 3238/3552
Test Top-1 accuracy: 91.16%
Test Top-5 accuracy: 99.13%
Test Macro F1: 0.9118
Test Micro F1: 0.9119
Test Weighted F1: 0.9104
Mean per-class accuracy: 0.9144
Training time: 06m 40s

Comparison with BiGRU + Attention Baseline

The BiGRU + Attention model achieved stronger performance on the same ASL Citizen processed dataset:

text

Metric                  BiGRU + Attention      Small Transformer
Validation Top-1        85.47%               78.13%
Test Top-1              98.48%               91.16%
Test Top-5              99.58%               99.13%
Test Macro F1           0.9850               0.9118
Test Weighted F1        0.9848               0.9104
Training Time           26m 25s              06m 40s

Interpretation

The Small Transformer trained successfully, but it underperformed the BiGRU + Attention model.

The best validation Top-1 accuracy was:

text

BiGRU + Attention: 85.47%
Small Transformer: 78.13%

Therefore, the BiGRU + Attention encoder was selected as the stronger isolated sign encoder for future transfer into the continuous How2Sign CTC stage.

The Small Transformer is still useful as a comparison baseline for the thesis because it demonstrates that a self-attention temporal encoder was evaluated against a recurrent attention-based encoder.

Repository Files

text

asl_isolated_small_transformer.zip
checkpoints/best_small_transformer_model.pt
checkpoints/best_transformer_encoder_only.pt
checkpoints/last_small_transformer_model.pt
reports/final_metrics.json
reports/training_history.csv
reports/test_classification_report.csv
reports/test_classification_report.json
reports/test_confusion_matrix.npy
reports/test_per_class_accuracy.csv
metadata/label_to_id.json
metadata/id_to_label.json
training_config.json

Important Files

Full model checkpoint

checkpoints/best_small_transformer_model.pt

Use this file if you want to reload the complete Small Transformer classifier for isolated sign prediction.

Encoder-only checkpoint

checkpoints/best_transformer_encoder_only.pt

Use this file if you want to reuse the Transformer encoder in future experiments.

Full ZIP archive

asl_isolated_small_transformer.zip

This archive contains the full training output folder, including checkpoints, reports, label maps, and configuration files.

Loading the Model in Kaggle

Example download code:

python

from huggingface_hub import hf_hub_download

repo_id = "SharoonArshad/asl-citizen-small-transformer-encoder-200"

model_path = hf_hub_download(
    repo_id=repo_id,
    repo_type="model",
    filename="checkpoints/best_small_transformer_model.pt"
)

encoder_path = hf_hub_download(
    repo_id=repo_id,
    repo_type="model",
    filename="checkpoints/best_transformer_encoder_only.pt"
)

print(model_path)
print(encoder_path)

Loading the Full ZIP Archive

python

from huggingface_hub import hf_hub_download
import zipfile
from pathlib import Path

repo_id = "SharoonArshad/asl-citizen-small-transformer-encoder-200"

zip_path = hf_hub_download(
    repo_id=repo_id,
    repo_type="model",
    filename="asl_isolated_small_transformer.zip"
)

extract_dir = Path("/kaggle/working/asl_isolated_small_transformer_loaded")
extract_dir.mkdir(parents=True, exist_ok=True)

with zipfile.ZipFile(zip_path, "r") as zip_ref:
    zip_ref.extractall(extract_dir)

print("Extracted to:", extract_dir)

Thesis Use

This model can be documented as the Transformer-based isolated sign recognition baseline.

Suggested thesis statement:

A Small Transformer Encoder was trained on the processed ASL Citizen keypoints-200 dataset for 200 isolated ASL classes. The model achieved 78.13% validation Top-1 accuracy and 91.16% test Top-1 accuracy. Although it trained faster than the BiGRU + Attention model, it achieved lower validation and test performance. Therefore, the BiGRU + Attention encoder was selected as the primary isolated sign encoder for future CTC-based continuous sign recognition.

Limitations

The model is trained on isolated signs, not continuous signing.
It recognizes 200 selected ASL classes from the processed ASL Citizen dataset.
It underperformed the BiGRU + Attention baseline on validation accuracy and test accuracy.
It does not yet perform ASL sentence translation.
A larger Transformer may require more data or stronger regularization.

Next Step

The next project stage is to use the stronger BiGRU + Attention encoder checkpoint as the pretrained visual encoder for a CTC-based How2Sign continuous sign recognition model.