Ishara is a deep learning model designed for accurate recognition of American Sign Language (ASL) fingerspelling. It is based on a hybrid architecture that combines Squeezeformer and Conformer blocks with Conv1D layers for efficient feature extraction from hand, face, and pose landmark data.
This model is a submission to the Google ASLFR Competition and achieves robust performance on character-level prediction tasks.
Model Description
Ishara processes sequences of normalized hand, face, and pose landmarks to predict fingerspelled words at the character level. The architecture is designed to handle temporal variability and missing data using a combination of:
Squeezeformer blocks: For efficient sequence modeling.
Conformer blocks: For enhanced feature extraction.
Conv1D layers: For initial temporal feature extraction.
The output predictions are character-level sequences optimized using Connectionist Temporal Classification (CTC) loss.
Dataset
The model was trained and evaluated on the dataset provided by the Google ASLFR Competition, which consists of:
Hand landmarks: 21 points each for left and right hands.
Face landmarks: 40 key points.
Pose landmarks: 10 key points.
Labels: Text sequences representing fingerspelled words.
Usage
Inference with TFLite
The model is available in TensorFlow Lite format for real-time inference. To use the model:
python
1import tensorflow as tf
23# Load the TFLite model4interpreter = tf.lite.Interpreter("model.tflite")5interpreter.allocate_tensors()67# Define input-output8input_details = interpreter.get_input_details()9output_details = interpreter.get_output_details()1011# Input a sequence of landmarks12input_data =...# Preprocessed input sequence13interpreter.set_tensor(input_details[0]['index'], input_data)14interpreter.invoke()1516# Get the prediction17output_data = interpreter.get_tensor(output_details[0]['index'])18print("Predicted Sequence:", output_data)
Training Workflow
You can replicate the training process using TensorFlow. The training loop is as follows:
python
1from model import get_model
23# Define the model4model = get_model(5 dim=256,6 num_conv_squeeze_blocks=2,7 num_conv_conform_blocks=2,8 kernel_sizes=[11,5,3],9 num_conv_per_block=3,10 dropout_rate=0.211)1213# Train the model14history = model.fit(15 train_dataset,16 validation_data=val_dataset,17 epochs=N_EPOCHS,18 callbacks=[validation_callback, lr_callback, WeightDecayCallback()]19)
Normalized Character Error Rate (CER): Quantifies the model's robustness.
Real-Time Inference Speed: Assessed on 1080p video inputs.
Results
Normalised Levenshtein Distance: [0.728]
Inference Speed: [200ms]
Model Size: [17.9 Mb]
Deployment
The model is optimized for deployment in real-time systems using TensorFlow Lite. This makes it suitable for integration into mobile and embedded systems for ASL recognition tasks.
