Source: CrisisMMD dataset (Alam et al., 2017)
✅Original Labels (8 classes from annotations):
Infrastructure and utility damage
Vehicle damage
Rescue, volunteering, or donation efforts
Affected individuals
Injured or dead people
Missing or found people
Other relevant information
Not humanitarian
✅Label Preprocessing (Class Merging):
Vehicle damage merged into Infrastructure and utility damage
Missing or found people merged into Affected individuals
Not humanitarian retained as a separate class
Removed very low-frequency categories (e.g., "Missing or found people" as a separate class)
✅Final Label Set (5 classes total):
Infrastructure and utility damage
Rescue, volunteering, or donation efforts
Affected individuals
Injured or dead people
Not humanitarian
✅Multimodal Consistency:
Selected only those posts where text and image annotations matched
Resulted in a total of 8,219 consistent samples:
Train set: 6,574 posts
Test set: 1,644 posts
✅ Preprocessing Done
Text:
Tokenized using BERT tokenizer (bert-base-uncased)
Extracted input_ids and attention_mask
Image:
Processed using ResNet-50
Extracted 2048-dimensional image features
The preprocessed data was saved in PyTorch .pt format:
train_human.pt and test_human.pt
Each contains: input_ids, attention_mask, image_vector, and label
✅ Model Architecture
A custom multimodal classifier that combines BERT and ResNet-50 outputs:
Component Details
Text Encoder BERT base (bert-base-uncased) – outputs pooler_output (768-d)
Image Encoder Pre-extracted ResNet-50 image features (2048-d)
Fusion Concatenation → FC layers → Softmax over 5 classes
Classifier Fully connected layers with BatchNorm, ReLU, Dropout
✅ Training Setup
Loss Function: CrossEntropyLoss
Optimizer: AdamW
Scheduler: StepLR (γ = 0.9)
Epochs Tried: 1, 3, 5, 8, 10
Batch Size: 16
Runtime: ~2 minutes 20 seconds per epoch on Google Colab (T4 GPU)
✅ Evaluation Metrics
Accuracy
Precision
Recall
F1 Score
✅ Metrics(epoch 3 with highest accuracy)
✅ Test Accuracy : 0.8820
✅ Precision : 0.6854
✅ Recall : 0.7176
✅ F1 Score : 0.7005
