Multimodal deep learning based on multiple correspondence analysis for disaster management

Samira Pouyanfar, Yudong Tao, Haiman Tian, Shu Ching Chen, Mei-Ling Shyu

Research output: Contribution to journalArticlepeer-review

7 Scopus citations

Abstract

The fast and explosive growth of digital data in social media and World Wide Web has led to numerous opportunities and research activities in multimedia big data. Among them, disaster management applications have attracted a lot of attention in recent years due to its impacts on society and government. This study targets content analysis and mining for disaster management. Specifically, a multimedia big data framework based on the advanced deep learning techniques is proposed. First, a video dataset of natural disasters is collected from YouTube. Then, two separate deep networks including a temporal audio model and a spatio-temporal visual model are presented to analyze the audio-visual modalities in video clips effectively. Thereafter, the results of both models are integrated using the proposed fusion model based on the Multiple Correspondence Analysis (MCA) algorithm which considers the correlations between data modalities and final classes. The proposed multimodal framework is evaluated on the collected disaster dataset and compared with several state-of-the-art single modality and fusion techniques. The results demonstrate the effectiveness of both visual model and fusion model compared to the baseline approaches. Specifically, the accuracy of the final multi-class classification using the proposed MCA-based fusion reaches to 73% on this challenging dataset.

Original languageEnglish (US)
JournalWorld Wide Web
DOIs
StateAccepted/In press - Jan 1 2018

Keywords

  • Disaster information management
  • Multimodal deep learning
  • Multiple Correspondence Analysis (MCA)

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Multimodal deep learning based on multiple correspondence analysis for disaster management'. Together they form a unique fingerprint.

Cite this