Processing social media images by combining human and machine intelligence during crises

What is it about?

The use of social media such as Twitter and Facebook at times of natural or man-made disasters has increased recently. Studies show the significance and usefulness of this online information for humanitarian organizations struggling with disaster response and management. A majority of these studies have however been relying almost exclusively on textual content (i.e., posts, messages, tweets, etc.) for crisis response and management tasks. Contrary (or complementary) to the existing literature on using social media textual content for crisis management, this work focuses on leveraging the social media visual content (i.e., images) to show humanitarian organizations its utility for disaster response.

Featured Image

Why is it important?

One of the important challenges of using social media data as a datasource is that a large proportion of images shared on social media is redundant or irrelevant, which requires robust filtering mechanisms. Another important challenge is that images acquired after major disasters do not share the same characteristics as those in large-scale image collections with clean annotations of well-defined object categories such as house, car, airplane, cat, dog, etc., used traditionally in computer vision research. To tackle these challenges, we present a social media image processing pipeline that combines human and machine intelligence to perform two important tasks: (i) capturing and filtering of social media imagery content (i.e., real-time image streaming, de-duplication, and relevancy filtering); and (ii) actionable information extraction (i.e., damage severity assessment) as a core situational awareness task during an on-going crisis event. Results obtained from extensive experiments on real-world crisis datasets demonstrate the significance of the proposed pipeline for optimal utilization of both human and machine computing resources.

Perspectives