Disasterlens v2 Multimodal Disaster Response

--- license: mit tags: - computer-vision - transfer-learning - disaster-response - multimodal - image-classification - text-classification - convnext - distilbert - late-fusion - crisis-informatics pipeline_tag: image-classification --- # DisasterLens V2: Multimodal Disaster-Response Classifier DisasterLens V2 is a two-stage multimodal disaster-response classifier built using computer vision, transfer learning, NLP, and late probability fusion. It analyzes a disaster-related image and optional social media text to predict: 1. Whether the post is informative for disaster response. 2. If informative, the humanitarian category. ## Architecture ### Stage 1: Informativeness Detection Predicts: - `informative` - `not_informative` Models: - ConvNeXt-Tiny image classifier with test-time augmentation - DistilBERT text classifier - Late fusion when text is available Fusion weights: - image: `0.55` - text: `0.45` ### Stage 2: Humanitarian Category Classification Predicts one of 5 grouped humanitarian categories: - `not_humanitarian` - `damage` - `rescue_or_donation` - `affected_people` - `other_relevant_information` Models: - ConvNeXt-Tiny image classifier with test-time augmentation - DistilBERT text classifier - Late fusion when text is available Fusion weights: - image: `0.65` - text: `0.35` ## Final Results | Component | Input | Accuracy | F1 Type | F1 | |---|---|---:|---|---:| | Informativeness image-only | Image | 0.8494 | Binary F1 | 0.8565 | | Informativeness fusion | Image + Text | 0.8632 | Binary F1 | 0.8701 | | Humanitarian image-only | Image | 0.7948 | Macro F1 | 0.7375 | | Humanitarian fusion | Image + Text | 0.7993 | Macro F1 | 0.7436 | | Full V2 cascade | Image + Text | 0.8485 | Macro F1 | 0.7837 | Final full V2 cascade: - Accuracy: **84.85%** - Macro F1: **78.37%** - Weighted F1: **84.44%** ## Usage Run from project root: ```bash python scripts/34_predict_disasterlens_v2.py \ --image_path "path/to/disaster_image.jpg" \ --text "optional tweet or post text" \ --json ``` Image only mode: ```bash python scripts/34_predict_disasterlens_v2.py \ --image_path "path/to/disaster_image.jpg" \ --json ``` Example Output: ```json { "mode": "multimodal", "informativeness": { "prediction": "informative", "confidence": 0.9507 }, "humanitarian": { "mode": "image_text_fusion", "prediction": "other_relevant_information", "confid...