Indian Sign Language Detection with IoT Integration

Features

• Real-time hand gesture recognition
• Support for 36 different signs (A-Z, 0-9)
• Word suggestions based on detected gestures
• IoT integration with ESP32-CAM
• OLED display output
• Support for both single and double-handed gestures
• ~98% accuracy on test data

System Architecture

1. Image Capture:
   • ESP32-CAM captures images
   • Sends via HTTP POST to Flask server
2. Processing Server:
   • Flask server receives images
   • Processes using MediaPipe for hand landmark detection
   • Uses TensorFlow model for gesture recognition
3. Display:
   • Results sent back to ESP32
   • Displayed on SSD1306 OLED screen
   • Shows gesture and confidence score

Requirements

Software

• Python 3.8 or higher
• Arduino IDE for ESP32
• Required Python packages (see requirements.txt)

Hardware

• ESP32-CAM module
• SSD1306 OLED Display (128x64)
• USB-TTL converter for ESP32 programming
• Connecting wires
• 5V power supply

Installation

1. Clone the repository:

1. Create and activate a virtual environment:

1. Install required packages:

1. Configure ESP32:
   • Open esp32_code/esp32_cam_oled.ino in Arduino IDE
   • Install required libraries:
     • ESP32 board support
     • Adafruit SSD1306
     • Adafruit GFX
   • Update WiFi credentials and server IP
   • Upload to ESP32-CAM

Project Structure

ISL-sign-lang-detection/
├── flask_server/
│   ├── app.py              # Flask server
│   ├── static/             # Static files
│   ├── templates/          # HTML templates
│   └── test_client.py      # Test script
├── esp32_code/
│   └── esp32_cam_oled.ino  # ESP32 code
├── dummy_input/            # Test images
├── gesture_model.h5        # Trained model
├── train.py               # Training script
├── detect.py              # Webcam detection script
├── process.py             # Data processing
└── README.md

Usage

Server Setup

1. Start the Flask server:

Testing Without Hardware

1. Add test images to dummy_input/ directory
2. Run the test client:

Hardware Setup

1. Connect OLED display to ESP32-CAM:
   • VCC → 5V
   • GND → GND
   • SCL → GPIO22
   • SDA → GPIO21
2. Power up the ESP32-CAM
3. The system will automatically:
   • Connect to WiFi
   • Start capturing images
   • Display results on OLED

Model Architecture

• Input layer: Hand landmark features
• Hidden layers with dropout and batch normalization
• Output layer: 36 classes (A-Z, 0-9)

Performance

• Training accuracy: ~98%
• Real-time detection with confidence scores
• Gesture stability checking to prevent false detections

Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

Acknowledgments

• TensorFlow team for the deep learning framework
• MediaPipe team for the hand landmark detection system
• OpenCV team for the computer vision tools
• ESP32 community for IoT support