KairoAI is an Indian Sign Language (ISL) learning application designed specifically for children. The app uses real-time hand gesture detection via the device camera to teach ISL alphabets and words, providing instant feedback to students.
Core Innovation
The app combines three powerful technologies:
Flutter for cross-platform UI
MediaPipe for hand detection (running natively on Android)
TensorFlow Lite for sign language classification
Key Differentiator
Unlike traditional learning apps, KairoAI provides real-time visual feedback by:
Showing the user what sign to make
Detecting their hand position using the camera
Validating if they're making the correct sign
Providing instant feedback (success/try again)
2. Project Vision & Goals
Primary Goal
Create an accessible, engaging platform for children to learn Indian Sign Language through interactive, AI-powered lessons.
Target Users
Primary: Children aged 6-14 learning ISL
Secondary: Parents and educators teaching ISL
Tertiary: Anyone interested in learning ISL
Core Features
1. Lesson Mode
Display a target alphabet (e.g., "A") or word (e.g., "MEGH")
Camera Image → MediaPipe (find hand) → Extract landmarks → DNN Model → Letter
Benefit: Fast, small dataset, background-independent
Why Two AI Models?
Model 1: MediaPipe Hands (Google's Pre-trained Model)
Job: Find the hand and identify 21 key points
Input: Camera frame (640×480 pixels)
Output: 21 landmark points (x, y, z coordinates)
Example landmarks:
Point 0: Wrist
Point 1-4: Thumb (base to tip)
Point 5-8: Index finger
Point 9-12: Middle finger
Point 13-16: Ring finger
Point 17-20: Pinky finger
Why use it?
Already trained by Google on millions of images
Works in real-time on mobile devices
Handles different hand sizes, skin tones, lighting
Free to use
Model 2: Your Custom TFLite Model (Train Yourself)
Job: Classify the 21 landmark points into ISL letters
MediaPipe is Google's open-source framework for building multimodal (video, audio, text) applied machine learning pipelines.
MediaPipe Hands Solution
Specifically designed to detect and track hands in real-time.
Key Capabilities:
Detects up to 2 hands simultaneously
Works in various lighting conditions
Handles different hand sizes and skin tones
Runs efficiently on mobile devices
Provides 21 3D landmark points per hand
The 21 Hand Landmarks
Landmark Numbering System:
8 12 16 20
│ │ │ │
7───11──15──19──│
│ │ │ │ │
6───10──14──18──│
│ │ │ │ │
5───9───13──17──┘
\
4───3───2───1
\
0
Point 0: Wrist
Point 1: Thumb CMC (base)
Point 2: Thumb MCP
Point 3: Thumb IP
Point 4: Thumb tip
Point 5: Index finger MCP
Point 6: Index finger PIP
Point 7: Index finger DIP
Point 8: Index finger tip
Point 9: Middle finger MCP
Point 10: Middle finger PIP
Point 11: Middle finger DIP
Point 12: Middle finger tip
Point 13: Ring finger MCP
Point 14: Ring finger PIP
Point 15: Ring finger DIP
Point 16: Ring finger tip
Point 17: Pinky MCP
Point 18: Pinky PIP
Point 19: Pinky DIP
Point 20: Pinky tip
Coordinate System
Each landmark has 3 coordinates:
X Coordinate
Range: 0.0 to 1.0
0.0 = left edge of image
1.0 = right edge of image
Normalized (independent of image resolution)
Y Coordinate
Range: 0.0 to 1.0
0.0 = top edge of image
1.0 = bottom edge of image
Normalized (independent of image resolution)
Z Coordinate
Approximate depth from wrist
Smaller values = closer to camera
Relative to wrist (Point 0)
Units: roughly in same scale as X
Example Coordinates
Letter "A" (closed fist with thumb up):
Point 0 (Wrist): x=0.50, y=0.70, z=0.00
Point 1 (Thumb base): x=0.48, y=0.65, z=0.02
Point 2 (Thumb mid): x=0.46, y=0.58, z=0.03
Point 3 (Thumb bend): x=0.44, y=0.52, z=0.04
Point 4 (Thumb tip): x=0.42, y=0.45, z=0.05
Point 5 (Index base): x=0.54, y=0.66, z=0.01
Point 6 (Index mid): x=0.56, y=0.68, z=0.00
Point 7 (Index bend): x=0.57, y=0.69, z=-0.01
Point 8 (Index tip): x=0.58, y=0.70, z=-0.02
...
MediaPipe Integration Code
kotlin
1// File: android/app/src/main/kotlin/com/kairo/ai/ml/HandLandmarkDetector.kt23import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker
4import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarkerResult
5import com.google.mediapipe.framework.image.BitmapImageBuilder
6import com.google.mediapipe.framework.image.MPImage
7import android.graphics.Bitmap
8import android.content.Context
910classHandLandmarkDetector(context: Context){1112privateval handLandmarker: HandLandmarker
1314init{15// Configure MediaPipe Hands16val options = HandLandmarker.HandLandmarkerOptions.builder()17.setBaseOptions(18 BaseOptions.builder()19.setModelAssetPath("hand_landmarker.task")// MediaPipe's pre-trained model20.build()21)22.setNumHands(1)// Detect only one hand23.setMinHandDetectionConfidence(0.5f)// 50% confidence threshold24.setMinHandPresenceConfidence(0.5f)25.setMinTrackingConfidence(0.5f)26.build()2728 handLandmarker = HandLandmarker.createFromOptions(context, options)29}3031/**
32 * Detect hand landmarks from a camera frame
33 *
34 * @param bitmap The camera frame
35 * @return FloatArray of 63 values [x0,y0,z0, x1,y1,z1, ..., x20,y20,z20]
36 * or null if no hand detected
37 */38fundetectLandmarks(bitmap: Bitmap): FloatArray?{39// Convert Android Bitmap to MediaPipe Image40val mpImage: MPImage =BitmapImageBuilder(bitmap).build()4142// Run hand detection43val result: HandLandmarkerResult = handLandmarker.detect(mpImage)4445// Check if any hands were detected46if(result.landmarks().isEmpty()){47returnnull// No hand found48}4950// Get landmarks from first detected hand51val handLandmarks = result.landmarks()[0]5253// Convert to flat array54val landmarkArray =FloatArray(63)5556for(i in0 until 21){57val landmark = handLandmarks[i]58 landmarkArray[i *3+0]= landmark.x()59 landmarkArray[i *3+1]= landmark.y()60 landmarkArray[i *3+2]= landmark.z()61}6263return landmarkArray
64}6566/**
67 * Optional: Normalize landmarks relative to wrist
68 * This makes the model robust to hand position/size
69 */70funnormalizeLandmarks(landmarks: FloatArray): FloatArray {71val normalized =FloatArray(63)7273// Get wrist coordinates (point 0)74val wristX = landmarks[0]75val wristY = landmarks[1]76val wristZ = landmarks[2]7778// Normalize all points relative to wrist79for(i in0 until 21){80 normalized[i *3+0]= landmarks[i *3+0]- wristX
81 normalized[i *3+1]= landmarks[i *3+1]- wristY
82 normalized[i *3+2]= landmarks[i *3+2]- wristZ
83}8485return normalized
86}87}
8. DNN Model Explained
What is a DNN (Dense Neural Network)?
A DNN is a type of artificial neural network where every neuron in one layer is connected to every neuron in the next layer.
Simple Analogy
Think of it as a decision-making chain:
Your Brain Recognizing a Friend:
Eyes see features → Brain processes patterns → Brain decides who it is
(height, hair, (tall + brown hair ("It's John!")
glasses, voice) + glasses = pattern)
DNN for Sign Language
Input: 63 numbers → Hidden layers process → Output: Letter prediction
(hand landmarks) (find patterns) (A-Z with confidence)
1# Our DNN Model Architecture23Input Layer:4 Shape:(63,)5 Description:63 landmark coordinates
6 Example:[0.45,0.82,0.01,0.52,0.75,...]78 ↓ (fully connected)910Hidden Layer 1:11 Neurons:12812 Activation: ReLU (Rectified Linear Unit)13 Dropout:30%(prevents overfitting)14 Description: Learns basic patterns (finger positions)1516 ↓ (fully connected)1718Hidden Layer 2:19 Neurons:6420 Activation: ReLU
21 Dropout:30%22 Description: Learns complex patterns (hand shapes)2324 ↓ (fully connected)2526Hidden Layer 3:27 Neurons:3228 Activation: ReLU
29 Dropout:20%30 Description: Learns letter-specific features
3132 ↓ (fully connected)3334Output Layer:35 Neurons:2636 Activation: Softmax
37 Description: Probability for each letter (A-Z)38 Example Output:[0.01,0.03,0.95,0.00,...,0.01]39(1%3%95%0%1%)40 A B C D ... Z
What Each Layer Does
1. Input Layer (63 neurons)
Receives raw landmark coordinates
No processing, just passes data forward
2. Hidden Layer 1 (128 neurons)
Learns basic geometric relationships
Examples:
"Are fingers spread apart?"
"Is thumb extended?"
"What's the palm orientation?"
3. Hidden Layer 2 (64 neurons)
Combines basic patterns into complex ones
Examples:
"Thumb up + fingers curled = might be 'A'"
"All fingers extended = might be 'B'"
4. Hidden Layer 3 (32 neurons)
Fine-tunes letter-specific features
Distinguishes similar signs
Examples:
"Is this 'M' or 'N'?" (very similar in ISL)
5. Output Layer (26 neurons)
Each neuron represents one letter
Softmax ensures probabilities sum to 1.0
Highest probability = predicted letter
Activation Functions
ReLU (Rectified Linear Unit)
Formula: f(x) = max(0, x)
Graph:
│ ╱
│ ╱
│ ╱
────┼─────
│
Benefits:
- Fast to compute
- Prevents vanishing gradients
- Works well for hidden layers
Softmax
Formula: softmax(xi) = e^xi / Σ(e^xj)
Purpose:
- Converts raw scores to probabilities
- All outputs sum to 1.0
- Used in output layer for classification
Example:
Raw scores: [2.1, 0.5, 4.2, 1.3]
After softmax: [0.12, 0.02, 0.84, 0.02]
(12% 2% 84% 2% )
Dropout Layers
Purpose: Prevent overfitting
During Training:
- Randomly "turn off" 30% of neurons
- Forces network to learn robust features
- Network can't rely on specific neurons
During Inference (app usage):
- All neurons active
- Uses learned patterns to classify
Model Training Code
python
1# File: model_training/train_model.py23import tensorflow as tf
4from tensorflow import keras
5import pandas as pd
6import numpy as np
7from sklearn.model_selection import train_test_split
8from sklearn.preprocessing import LabelEncoder
910# Load landmark dataset11print("Loading dataset...")12data = pd.read_csv('landmarks_dataset.csv')1314print(f"Dataset shape: {data.shape}")15print(f"Classes: {sorted(data['label'].unique())}")1617# Separate features (X) and labels (y)18X = data.iloc[:,:-1].values # 63 landmark columns19y = data.iloc[:,-1].values # Label column2021# Encode labels: A=0, B=1, C=2, ..., Z=2522encoder = LabelEncoder()23y_encoded = encoder.fit_transform(y)24y_categorical = keras.utils.to_categorical(y_encoded)2526# Save label mapping27label_mapping ={i: label for i, label inenumerate(encoder.classes_)}28print(f"Label mapping: {label_mapping}")2930# Train/test split (80% train, 20% test)31X_train, X_test, y_train, y_test = train_test_split(32 X, y_categorical,33 test_size=0.2,34 random_state=42,35 stratify=y_categorical # Maintain class distribution36)3738print(f"\nTraining samples: {len(X_train)}")39print(f"Testing samples: {len(X_test)}")4041# Build the DNN model42model = keras.Sequential([43# Input layer44 keras.layers.Input(shape=(63,), name='landmark_input'),4546# Hidden layer 147 keras.layers.Dense(128, activation='relu', name='dense_1'),48 keras.layers.BatchNormalization(),49 keras.layers.Dropout(0.3),5051# Hidden layer 252 keras.layers.Dense(64, activation='relu', name='dense_2'),53 keras.layers.BatchNormalization(),54 keras.layers.Dropout(0.3),5556# Hidden layer 357 keras.layers.Dense(32, activation='relu', name='dense_3'),58 keras.layers.Dropout(0.2),5960# Output layer61 keras.layers.Dense(len(encoder.classes_), activation='softmax', name='output')62])6364# Compile model65model.compile(66 optimizer=keras.optimizers.Adam(learning_rate=0.001),67 loss='categorical_crossentropy',68 metrics=['accuracy']69)7071# Display model summary72model.summary()7374# Training callbacks75callbacks =[76# Stop training if validation loss doesn't improve for 10 epochs77 keras.callbacks.EarlyStopping(78 monitor='val_loss',79 patience=10,80 restore_best_weights=True81),8283# Reduce learning rate if validation loss plateaus84 keras.callbacks.ReduceLROnPlateau(85 monitor='val_loss',86 factor=0.5,87 patience=5,88 min_lr=0.0000189),9091# Save best model during training92 keras.callbacks.ModelCheckpoint(93'best_model.h5',94 monitor='val_accuracy',95 save_best_only=True96)97]9899# Train the model100print("\nTraining model...")101history = model.fit(102 X_train, y_train,103 epochs=100,104 batch_size=32,105 validation_split=0.2,# Use 20% of training data for validation106 callbacks=callbacks,107 verbose=1108)109110# Evaluate on test set111print("\nEvaluating on test set...")112test_loss, test_accuracy = model.evaluate(X_test, y_test, verbose=0)113print(f"Test Loss: {test_loss:.4f}")114print(f"Test Accuracy: {test_accuracy*100:.2f}%")115116# Save final model117model.save('isl_model.h5')118print("\nModel saved as 'isl_model.h5'")119120# Convert to TFLite121print("\nConverting to TFLite...")122converter = tf.lite.TFLiteConverter.from_keras_model(model)123converter.optimizations =[tf.lite.Optimize.DEFAULT]124tflite_model = converter.convert()125126withopen('isl_model.tflite','wb')as f:127 f.write(tflite_model)128129print(f"TFLite model saved!")130print(f"Model size: {len(tflite_model)/1024:.2f} KB")
Epoch 1/100
────────────────────────────────────────
Batch 1/250: Forward pass → Calculate loss → Backward pass → Update weights
Batch 2/250: Forward pass → Calculate loss → Backward pass → Update weights
...
Batch 250/250: Forward pass → Calculate loss → Backward pass → Update weights
Validation:
- Test on validation set (20% of training data)
- Calculate validation accuracy
- If improved, save as best model
Epoch 1: loss=0.5423, accuracy=0.8234, val_loss=0.4321, val_accuracy=0.8567
Epoch 2: loss=0.3215, accuracy=0.8876, val_loss=0.3102, val_accuracy=0.8923
Epoch 3: loss=0.2543, accuracy=0.9123, val_loss=0.2876, val_accuracy=0.9034
...
Epoch 45: loss=0.0234, accuracy=0.9892, val_loss=0.0456, val_accuracy=0.9845
Epoch 46: loss=0.0231, accuracy=0.9894, val_loss=0.0458, val_accuracy=0.9844
(No improvement for 10 epochs → Early stopping triggered)
Best model: Epoch 45 with val_accuracy=0.9845
9. Platform Channels Explained
What Are Platform Channels?
Platform channels are Flutter's official mechanism for communication between Dart code and native platform code (Kotlin/Swift).
The Problem They Solve
Flutter (Dart) runs in its own runtime
↓
Cannot directly access:
- Native camera APIs
- MediaPipe library (Android/iOS)
- Hardware sensors
- Native ML frameworks
- Bluetooth, NFC, etc.
Solution: Platform Channels = Bridge between worlds
Types of Platform Channels
1. MethodChannel (Request-Response)
Use Case: One-time requests with responses
Flutter asks → Kotlin does work → Kotlin responds → Flutter receives
Example: Start/stop camera, take photo, get device info
dart
1// Flutter side2final result =await methodChannel.invokeMethod('getCameraStatus');3print(result);// "active" or "inactive"
kotlin
1// Kotlin side2methodChannel.setMethodCallHandler{ call, result ->3when(call.method){4"getCameraStatus"->{5val status =if(cameraActive)"active"else"inactive"6 result.success(status)7}8}9}
2. EventChannel (Continuous Stream)
Use Case: Continuous data stream from native to Flutter
1// Flutter side - Handling errors23try{4await signDetectionService.startDetection();5}onPlatformExceptioncatch(e){6switch(e.code){7case'CAMERA_ERROR':8showSnackBar('Camera failed to start');9break;10case'PERMISSION_DENIED':11showSnackBar('Camera permission required');12break;13case'MEDIAPIPE_ERROR':14showSnackBar('Hand detection failed');15break;16default:17showSnackBar('Unknown error: ${e.message}');18}19}
kotlin
1// Kotlin side - Sending errors23try{4startCamera()5 result.success(null)6}catch(e: SecurityException){7 result.error("PERMISSION_DENIED","Camera permission not granted",null)8}catch(e: CameraAccessException){9 result.error("CAMERA_ERROR","Failed to access camera: ${e.message}",null)10}catch(e: Exception){11 result.error("UNKNOWN_ERROR", e.message,null)12}
10. Dataset Creation Guide
Understanding the Dataset
What You Need
For a 26-letter ISL alphabet app, you need:
Dataset Size Calculation:
- 26 letters (A-Z)
- 500-1000 images per letter (recommended)
- Total: 13,000 - 26,000 images
Actual Data After Extraction:
- Each image → 1 row in CSV
- Each row = 63 landmark values + 1 label
- Final CSV: 13,000-26,000 rows × 64 columns
Dataset Quality Factors
Factor
Impact on Accuracy
Recommendation
Number of samples
High
500+ per letter
Variety of people
High
5-10 different people
Hand orientations
Medium
Multiple angles
Lighting conditions
Low (landmarks robust)
Normal indoor lighting OK
Background
None (landmarks only)
Any background works
Camera distance
Medium
Keep consistent (arm's length)
Option 1: Use Existing Dataset (Fastest)
Step 1: Find ISL Dataset
bash
1# Search on Kaggle2https://www.kaggle.com/search?q=indian+sign+language
34# Popular datasets:5# 1. "ISL Dataset" by various authors6# 2. "Indian Sign Language Recognition Dataset"7# 3. "ISL Alphabet Dataset"
Equipment Needed:
- Smartphone camera
- Good lighting (natural or indoor)
- Plain background (optional but helpful)
Process:
1. Record 30-second video per letter
2. Person makes the sign continuously
3. Vary hand position slightly
4. Extract frames → 200-300 images per video
Advantages:
- Quick data collection (30 min for all 26 letters)
- Natural hand movements
- Variety in positioning
Step-by-Step Video Collection
python
1# File: data_collection/extract_frames_from_video.py23import cv2
4import os
56defextract_frames_from_video(video_path, output_folder, letter, frame_interval=3):7"""
8 Extract frames from video at specified interval
910 Args:
11 video_path: Path to video file
12 output_folder: Where to save frames
13 letter: ISL letter (A-Z)
14 frame_interval: Extract every Nth frame (3 = every 3rd frame)
15 """16# Create output directory17 letter_folder = os.path.join(output_folder, letter)18 os.makedirs(letter_folder, exist_ok=True)1920# Open video21 cap = cv2.VideoCapture(video_path)2223 frame_count =024 saved_count =02526print(f"Processing video: {video_path}")2728whileTrue:29 ret, frame = cap.read()3031ifnot ret:32break3334# Extract every Nth frame35if frame_count % frame_interval ==0:36 output_path = os.path.join(37 letter_folder,38f"{letter}_{saved_count:04d}.jpg"39)40 cv2.imwrite(output_path, frame)41 saved_count +=14243 frame_count +=14445 cap.release()4647print(f"✅ Extracted {saved_count} frames for letter '{letter}'")48print(f" Saved to: {letter_folder}")4950# Usage51if __name__ =="__main__":52# Extract frames from all videos53 videos =[54("videos/letter_A.mp4","A"),55("videos/letter_B.mp4","B"),56# ... add all 26 letters57]5859 output_folder ="extracted_frames"6061for video_path, letter in videos:62 extract_frames_from_video(video_path, output_folder, letter, frame_interval=3)6364print("\n✅ All frames extracted!")
Method 2: Photo Collection App
python
1# File: data_collection/photo_collector.py23import cv2
4import os
5import time
67defcollect_photos_for_letter(letter, num_photos=500):8"""
9 Interactive photo collection using webcam
1011 Args:
12 letter: ISL letter to collect (A-Z)
13 num_photos: Number of photos to capture
14 """15# Create output directory16 output_folder =f"collected_data/{letter}"17 os.makedirs(output_folder, exist_ok=True)1819# Open webcam20 cap = cv2.VideoCapture(0)2122print<!-- filepath: d:\study files\FlutterProjects\KairoAI\DOCUMENTATION.md -->23# KairoAI - Complete Project Documentation24## Indian Sign Language Learning App with AI-Powered Hand Detection2526**Author:** Megh Modi
27**Created:** December 18,202528**Version:**1.0.029**Status:** Planning & Architecture Phase
3031---3233# Table of Contents34351.[Executive Summary](#executive-summary)362.[Project Vision & Goals](#project-vision--goals)373.[Technical Architecture](#technical-architecture)384.[Technology Stack](#technology-stack)395.[Understanding the AI Pipeline](#understanding-the-ai-pipeline)406.[Data Flow & Pipeline](#data-flow--pipeline)417.[MediaPipe Explained](#mediapipe-explained)428.[DNN Model Explained](#dnn-model-explained)439.[Platform Channels Explained](#platform-channels-explained)4410.[Dataset Creation Guide](#dataset-creation-guide)4511.[Model Training Guide](#model-training-guide)4612.[Implementation Roadmap](#implementation-roadmap)4713.[Code Structure](#code-structure)4814.[Challenges & Solutions](#challenges--solutions)4915.[Feasibility Assessment](#feasibility-assessment)5016.[Resources & Learning Path](#resources--learning-path)5152---5354# 1. Executive Summary5556## What is KairoAI?5758KairoAI is an Indian Sign Language (ISL) learning application designed specifically for children. The app uses real-time hand gesture detection via the device camera to teach ISL alphabets and words, providing instant feedback to students.5960## Core Innovation6162The app combines three powerful technologies:63-**Flutter**for cross-platform UI
64-**MediaPipe**for hand detection (running natively on Android)65-**TensorFlow Lite**for sign language classification
6667## Key Differentiator6869Unlike traditional learning apps, KairoAI provides **real-time visual feedback** by:701. Showing the user what sign to make
712. Detecting their hand position using the camera
723. Validating if they're making the correct sign
734. Providing instant feedback (success/try again)7475---7677# 2. Project Vision & Goals7879## Primary Goal8081Create an accessible, engaging platform for children to learn Indian Sign Language through interactive, AI-powered lessons.8283## Target Users8485-**Primary:** Children aged 6-14 learning ISL
86-**Secondary:** Parents and educators teaching ISL
87-**Tertiary:** Anyone interested in learning ISL
8889## Core Features9091### 1. Lesson Mode92- Display a target alphabet (e.g.,"A")or word (e.g.,"MEGH")93- Open device camera
94- Detect student's hand sign in real-time
95- Validate against expected sign
96- Show success animation/sound on correct detection
97- Provide guidance hints on incorrect attempts
9899### 2. Quiz Mode100- Present random alphabets or words
101- Student performs signs sequentially
102- Each detected letter is validated in order
103- Progress only on correct detection
104- Track accuracy and completion time
105106### 3. Progress Tracking107- Store lesson completion in Firebase Firestore
108- Track quiz scores and accuracy
109- Visualize learning progress over time
110- Gamification elements (badges, streaks)111112---113114# 3. Technical Architecture115116## High-Level Architecture117
Camera Image → MediaPipe (find hand) → Extract landmarks → DNN Model → Letter
Benefit: Fast, small dataset, background-independent
Why Two AI Models?
Model 1: MediaPipe Hands (Google's Pre-trained Model)
Job: Find the hand and identify 21 key points
Input: Camera frame (640×480 pixels)
Output: 21 landmark points (x, y, z coordinates)
Example landmarks:
Point 0: Wrist
Point 1-4: Thumb (base to tip)
Point 5-8: Index finger
Point 9-12: Middle finger
Point 13-16: Ring finger
Point 17-20: Pinky finger
Why use it?
Already trained by Google on millions of images
Works in real-time on mobile devices
Handles different hand sizes, skin tones, lighting
Free to use
Model 2: Your Custom TFLite Model (Train Yourself)
Job: Classify the 21 landmark points into ISL letters
MediaPipe is Google's open-source framework for building multimodal (video, audio, text) applied machine learning pipelines.
MediaPipe Hands Solution
Specifically designed to detect and track hands in real-time.
Key Capabilities:
Detects up to 2 hands simultaneously
Works in various lighting conditions
Handles different hand sizes and skin tones
Runs efficiently on mobile devices
Provides 21 3D landmark points per hand
The 21 Hand Landmarks
Landmark Numbering System:
8 12 16 20
│ │ │ │
7───11──15──19──│
│ │ │ │ │
6───10──14──18──│
│ │ │ │ │
5───9───13──17──┘
\
4───3───2───1
\
0
Point 0: Wrist
Point 1: Thumb CMC (base)
Point 2: Thumb MCP
Point 3: Thumb IP
Point 4: Thumb tip
Point 5: Index finger MCP
Point 6: Index finger PIP
Point 7: Index finger DIP
Point 8: Index finger tip
Point 9: Middle finger MCP
Point 10: Middle finger PIP
Point 11: Middle finger DIP
Point 12: Middle finger tip
Point 13: Ring finger MCP
Point 14: Ring finger PIP
Point 15: Ring finger DIP
Point 16: Ring finger tip
Point 17: Pinky MCP
Point 18: Pinky PIP
Point 19: Pinky DIP
Point 20: Pinky tip
Coordinate System
Each landmark has 3 coordinates:
X Coordinate
Range: 0.0 to 1.0
0.0 = left edge of image
1.0 = right edge of image
Normalized (independent of image resolution)
Y Coordinate
Range: 0.0 to 1.0
0.0 = top edge of image
1.0 = bottom edge of image
Normalized (independent of image resolution)
Z Coordinate
Approximate depth from wrist
Smaller values = closer to camera
Relative to wrist (Point 0)
Units: roughly in same scale as X
Example Coordinates
Letter "A" (closed fist with thumb up):
Point 0 (Wrist): x=0.50, y=0.70, z=0.00
Point 1 (Thumb base): x=0.48, y=0.65, z=0.02
Point 2 (Thumb mid): x=0.46, y=0.58, z=0.03
Point 3 (Thumb bend): x=0.44, y=0.52, z=0.04
Point 4 (Thumb tip): x=0.42, y=0.45, z=0.05
Point 5 (Index base): x=0.54, y=0.66, z=0.01
Point 6 (Index mid): x=0.56, y=0.68, z=0.00
Point 7 (Index bend): x=0.57, y=0.69, z=-0.01
Point 8 (Index tip): x=0.58, y=0.70, z=-0.02
...
MediaPipe Integration Code
kotlin
1// File: android/app/src/main/kotlin/com/kairo/ai/ml/HandLandmarkDetector.kt23import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarker
4import com.google.mediapipe.tasks.vision.handlandmarker.HandLandmarkerResult
5import com.google.mediapipe.framework.image.BitmapImageBuilder
6import com.google.mediapipe.framework.image.MPImage
7import android.graphics.Bitmap
8import android.content.Context
910classHandLandmarkDetector(context: Context){1112privateval handLandmarker: HandLandmarker
1314init{15// Configure MediaPipe Hands16val options = HandLandmarker.HandLandmarkerOptions.builder()17.setBaseOptions(18 BaseOptions.builder()19.setModelAssetPath("hand_landmarker.task")// MediaPipe's pre-trained model20.build()21)22.setNumHands(1)// Detect only one hand23.setMinHandDetectionConfidence(0.5f)// 50% confidence threshold24.setMinHandPresenceConfidence(0.5f)25.setMinTrackingConfidence(0.5f)26.build()2728 handLandmarker = HandLandmarker.createFromOptions(context, options)29}3031/**
32 * Detect hand landmarks from a camera frame
33 *
34 * @param bitmap The camera frame
35 * @return FloatArray of 63 values [x0,y0,z0, x1,y1,z1, ..., x20,y20,z20]
36 * or null if no hand detected
37 */38fundetectLandmarks(bitmap: Bitmap): FloatArray?{39// Convert Android Bitmap to MediaPipe Image40val mpImage: MPImage =BitmapImageBuilder(bitmap).build()4142// Run hand detection43val result: HandLandmarkerResult = handLandmarker.detect(mpImage)4445// Check if any hands were detected46if(result.landmarks().isEmpty()){47returnnull// No hand found48}4950// Get landmarks from first detected hand51val handLandmarks = result.landmarks()[0]5253// Convert to flat array54val landmarkArray =FloatArray(63)5556for(i in0 until 21){57val landmark = handLandmarks[i]58 landmarkArray[i *3+0]= landmark.x()59 landmarkArray[i *3+1]= landmark.y()60 landmarkArray[i *3+2]= landmark.z()61}6263return landmarkArray
64}6566/**
67 * Optional: Normalize landmarks relative to wrist
68 * This makes the model robust to hand position/size
69 */70funnormalizeLandmarks(landmarks: FloatArray): FloatArray {71val normalized =FloatArray(63)7273// Get wrist coordinates (point 0)74val wristX = landmarks[0]75val wristY = landmarks[1]76val wristZ = landmarks[2]7778// Normalize all points relative to wrist79for(i in0 until 21){80 normalized[i *3+0]= landmarks[i *3+0]- wristX
81 normalized[i *3+1]= landmarks[i *3+1]- wristY
82 normalized[i *3+2]= landmarks[i *3+2]- wristZ
83}8485return normalized
86}87}
8. DNN Model Explained
What is a DNN (Dense Neural Network)?
A DNN is a type of artificial neural network where every neuron in one layer is connected to every neuron in the next layer.
Simple Analogy
Think of it as a decision-making chain:
Your Brain Recognizing a Friend:
Eyes see features → Brain processes patterns → Brain decides who it is
(height, hair, (tall + brown hair ("It's John!")
glasses, voice) + glasses = pattern)
DNN for Sign Language
Input: 63 numbers → Hidden layers process → Output: Letter prediction
(hand landmarks) (find patterns) (A-Z with confidence)
1# Our DNN Model Architecture23Input Layer:4 Shape:(63,)5 Description:63 landmark coordinates
6 Example:[0.45,0.82,0.01,0.52,0.75,...]78 ↓ (fully connected)910Hidden Layer 1:11 Neurons:12812 Activation: ReLU (Rectified Linear Unit)13 Dropout:30%(prevents overfitting)14 Description: Learns basic patterns (finger positions)1516 ↓ (fully connected)1718Hidden Layer 2:19 Neurons:6420 Activation: ReLU
21 Dropout:30%22 Description: Learns complex patterns (hand shapes)2324 ↓ (fully connected)2526Hidden Layer 3:27 Neurons:3228 Activation: ReLU
29 Dropout:20%30 Description: Learns letter-specific features
3132 ↓ (fully connected)3334Output Layer:35 Neurons:2636 Activation: Softmax
37 Description: Probability for each letter (A-Z)38 Example Output:[0.01,0.03,0.95,0.00,...,0.01]39(1%3%95%0%1%)40 A B C D ... Z
What Each Layer Does
1. Input Layer (63 neurons)
Receives raw landmark coordinates
No processing, just passes data forward
2. Hidden Layer 1 (128 neurons)
Learns basic geometric relationships
Examples:
"Are fingers spread apart?"
"Is thumb extended?"
"What's the palm orientation?"
3. Hidden Layer 2 (64 neurons)
Combines basic patterns into complex ones
Examples:
"Thumb up + fingers curled = might be 'A'"
"All fingers extended = might be 'B'"
4. Hidden Layer 3 (32 neurons)
Fine-tunes letter-specific features
Distinguishes similar signs
Examples:
"Is this 'M' or 'N'?" (very similar in ISL)
5. Output Layer (26 neurons)
Each neuron represents one letter
Softmax ensures probabilities sum to 1.0
Highest probability = predicted letter
Activation Functions
ReLU (Rectified Linear Unit)
Formula: f(x) = max(0, x)
Graph:
│ ╱
│ ╱
│ ╱
────┼─────
│
Benefits:
- Fast to compute
- Prevents vanishing gradients
- Works well for hidden layers
Softmax
Formula: softmax(xi) = e^xi / Σ(e^xj)
Purpose:
- Converts raw scores to probabilities
- All outputs sum to 1.0
- Used in output layer for classification
Example:
Raw scores: [2.1, 0.5, 4.2, 1.3]
After softmax: [0.12, 0.02, 0.84, 0.02]
(12% 2% 84% 2% )
Dropout Layers
Purpose: Prevent overfitting
During Training:
- Randomly "turn off" 30% of neurons
- Forces network to learn robust features
- Network can't rely on specific neurons
During Inference (app usage):
- All neurons active
- Uses learned patterns to classify
Model Training Code
python
1# File: model_training/train_model.py23import tensorflow as tf
4from tensorflow import keras
5import pandas as pd
6import numpy as np
7from sklearn.model_selection import train_test_split
8from sklearn.preprocessing import LabelEncoder
910# Load landmark dataset11print("Loading dataset...")12data = pd.read_csv('landmarks_dataset.csv')1314print(f"Dataset shape: {data.shape}")15print(f"Classes: {sorted(data['label'].unique())}")1617# Separate features (X) and labels (y)18X = data.iloc[:,:-1].values # 63 landmark columns19y = data.iloc[:,-1].values # Label column2021# Encode labels: A=0, B=1, C=2, ..., Z=2522encoder = LabelEncoder()23y_encoded = encoder.fit_transform(y)24y_categorical = keras.utils.to_categorical(y_encoded)2526# Save label mapping27label_mapping ={i: label for i, label inenumerate(encoder.classes_)}28print(f"Label mapping: {label_mapping}")2930# Train/test split (80% train, 20% test)31X_train, X_test, y_train, y_test = train_test_split(32 X, y_categorical,33 test_size=0.2,34 random_state=42,35 stratify=y_categorical # Maintain class distribution36)3738print(f"\nTraining samples: {len(X_train)}")39print(f"Testing samples: {len(X_test)}")4041# Build the DNN model42model = keras.Sequential([43# Input layer44 keras.layers.Input(shape=(63,), name='landmark_input'),4546# Hidden layer 147 keras.layers.Dense(128, activation='relu', name='dense_1'),48 keras.layers.BatchNormalization(),49 keras.layers.Dropout(0.3),5051# Hidden layer 252 keras.layers.Dense(64, activation='relu', name='dense_2'),53 keras.layers.BatchNormalization(),54 keras.layers.Dropout(0.3),5556# Hidden layer 357 keras.layers.Dense(32, activation='relu', name='dense_3'),58 keras.layers.Dropout(0.2),5960# Output layer61 keras.layers.Dense(len(encoder.classes_), activation='softmax', name='output')62])6364# Compile model65model.compile(66 optimizer=keras.optimizers.Adam(learning_rate=0.001),67 loss='categorical_crossentropy',68 metrics=['accuracy']69)7071# Display model summary72model.summary()7374# Training callbacks75callbacks =[76# Stop training if validation loss doesn't improve for 10 epochs77 keras.callbacks.EarlyStopping(78 monitor='val_loss',79 patience=10,80 restore_best_weights=True81),8283# Reduce learning rate if validation loss plateaus84 keras.callbacks.ReduceLROnPlateau(85 monitor='val_loss',86 factor=0.5,87 patience=5,88 min_lr=0.0000189),9091# Save best model during training92 keras.callbacks.ModelCheckpoint(93'best_model.h5',94 monitor='val_accuracy',95 save_best_only=True96)97]9899# Train the model100print("\nTraining model...")101history = model.fit(102 X_train, y_train,103 epochs=100,104 batch_size=32,105 validation_split=0.2,# Use 20% of training data for validation106 callbacks=callbacks,107 verbose=1108)109110# Evaluate on test set111print("\nEvaluating on test set...")112test_loss, test_accuracy = model.evaluate(X_test, y_test, verbose=0)113print(f"Test Loss: {test_loss:.4f}")114print(f"Test Accuracy: {test_accuracy*100:.2f}%")115116# Save final model117model.save('isl_model.h5')118print("\nModel saved as 'isl_model.h5'")119120# Convert to TFLite121print("\nConverting to TFLite...")122converter = tf.lite.TFLiteConverter.from_keras_model(model)123converter.optimizations =[tf.lite.Optimize.DEFAULT]124tflite_model = converter.convert()125126withopen('isl_model.tflite','wb')as f:127 f.write(tflite_model)128129print(f"TFLite model saved!")130print(f"Model size: {len(tflite_model)/1024:.2f} KB")
Epoch 1/100
────────────────────────────────────────
Batch 1/250: Forward pass → Calculate loss → Backward pass → Update weights
Batch 2/250: Forward pass → Calculate loss → Backward pass → Update weights
...
Batch 250/250: Forward pass → Calculate loss → Backward pass → Update weights
Validation:
- Test on validation set (20% of training data)
- Calculate validation accuracy
- If improved, save as best model
Epoch 1: loss=0.5423, accuracy=0.8234, val_loss=0.4321, val_accuracy=0.8567
Epoch 2: loss=0.3215, accuracy=0.8876, val_loss=0.3102, val_accuracy=0.8923
Epoch 3: loss=0.2543, accuracy=0.9123, val_loss=0.2876, val_accuracy=0.9034
...
Epoch 45: loss=0.0234, accuracy=0.9892, val_loss=0.0456, val_accuracy=0.9845
Epoch 46: loss=0.0231, accuracy=0.9894, val_loss=0.0458, val_accuracy=0.9844
(No improvement for 10 epochs → Early stopping triggered)
Best model: Epoch 45 with val_accuracy=0.9845
9. Platform Channels Explained
What Are Platform Channels?
Platform channels are Flutter's official mechanism for communication between Dart code and native platform code (Kotlin/Swift).
The Problem They Solve
Flutter (Dart) runs in its own runtime
↓
Cannot directly access:
- Native camera APIs
- MediaPipe library (Android/iOS)
- Hardware sensors
- Native ML frameworks
- Bluetooth, NFC, etc.
Solution: Platform Channels = Bridge between worlds
Types of Platform Channels
1. MethodChannel (Request-Response)
Use Case: One-time requests with responses
Flutter asks → Kotlin does work → Kotlin responds → Flutter receives
Example: Start/stop camera, take photo, get device info
dart
1// Flutter side2final result =await methodChannel.invokeMethod('getCameraStatus');3print(result);// "active" or "inactive"
kotlin
1// Kotlin side2methodChannel.setMethodCallHandler{ call, result ->3when(call.method){4"getCameraStatus"->{5val status =if(cameraActive)"active"else"inactive"6 result.success(status)7}8}9}
2. EventChannel (Continuous Stream)
Use Case: Continuous data stream from native to Flutter
1val result = handLandmarker.detect(mpImage)2println("Hands detected: ${result.landmarks().size}")34if(result.landmarks().isEmpty()){5println("❌ No hand detected")6println("Try: better lighting, move hand closer, show full hand")7}
Challenge 2: Low Model Accuracy
Symptoms:
Test accuracy < 85%
Wrong letter predictions
Low confidence scores
Solutions:
Collect More Data
python
1# Aim for at least 500 samples per letter2# Current: 100 per letter → Low3# Target: 500+ per letter → Good
1// Check if model file exists and loads correctly2try{3val modelFile =loadModelFile(context,"isl_model.tflite")4println("✅ Model loaded: ${modelFile.capacity()} bytes")5}catch(e: Exception){6println("❌ Failed to load model: ${e.message}")7}
Common Pitfalls
Pitfall 1: Not Normalizing Landmarks
Problem: Model accuracy drops when user changes hand position or distance
Solution: Always normalize landmarks during both training and inference
python
1# Training time2defnormalize_landmarks(landmarks):3# Make relative to wrist and scale to unit box4# ...56# Inference time (Kotlin)7fun normalizeLandmarks(landmarks: FloatArray): FloatArray {8// Same normalization logic
9//...10}
Pitfall 2: Imbalanced Dataset
Problem: Some letters have 1000 samples, others have 100
Solution: Balance the dataset
python
1# Check class distribution2print(data['label'].value_counts())34# Undersample majority classes or oversample minority classes5from imblearn.over_sampling import SMOTE
6X_balanced, y_balanced = SMOTE().fit_resample(X, y)
Pitfall 3: Forgetting to Close Camera
Problem: Camera stays on even after leaving screen, draining battery
1// Simple app that just shows hand landmarks2class MainActivity :AppCompatActivity(){3// Use MediaPipe to detect hand4// Draw landmarks on camera preview5// No classification yet6}
Practice Project 2: Platform Channel Hello World
Goal: Send message from Kotlin to Flutter
dart
1// Flutter2final result =await channel.invokeMethod('sayHello');3print(result);// "Hello from Kotlin!"
kotlin
1// Kotlin2channel.setMethodCallHandler{ call, result ->3if(call.method =="sayHello"){4 result.success("Hello from Kotlin!")5}6}
□ Check logs (Android Studio Logcat)
□ Verify dependencies versions match
□ Clean and rebuild project
□ Restart Android Studio
□ Check permissions in AndroidManifest.xml
□ Verify channel names match exactly
□ Test on real device (not emulator)
□ Check model file exists in assets
□ Verify input/output shapes
□ Print debug information at each step
✅ Technically feasible - All pieces exist and work
✅ Educationally valuable - You'll learn A LOT
✅ Portfolio-worthy - Impressive for job applications
✅ Challenging but doable - With persistence
This Project is NOT:
❌ A weekend project
❌ Impossible for beginners
❌ Requiring PhD-level ML knowledge
❌ Dependent on expensive tools
Success Factors
You WILL succeed if you:
✅ Start small (hand detection first, full app later)
✅ Break problems into tiny steps
✅ Debug systematically (logs everywhere)
✅ Ask for help when stuck (community is helpful)
✅ Accept imperfection (70% accuracy is a great start)
✅ Stay persistent (debugging takes time)
You might struggle if you:
❌ Try to do everything at once
❌ Skip the learning phase
❌ Give up at first error
❌ Aim for perfection immediately
Contact & Support
If you need help while building this:
GitHub Discussions - Most responsive
Stack Overflow - Tag your questions properly
Flutter Discord - Real-time chat
This AI Assistant - Come back anytime!
Conclusion
KairoAI is an ambitious but achievable project.
You have:
✅ Clear architecture
✅ Detailed implementation guide
✅ Code examples for every component
✅ Realistic timeline
✅ Troubleshooting guides
Now it's time to build!
Start with the MediaPipe example this week. Once you see hand detection working on your device, you'll realize this is not just possible—it's inevitable.
Good luck! 🚀
Last updated: December 18, 2025Version: 1.0.0Author: Megh Modi
Appendix: Quick Reference
Key Commands
bash
1# Flutter2flutter doctor
3flutter clean
4flutter pub get
5flutter run
67# Android8./gradlew clean
9./gradlew assembleDebug
1011# Python12pip install -r requirements.txt
13python extract_landmarks.py
14python train_model.py
