Multi-Task Multimodal Learning for Disaster Situation Assessment

Tianyi Wang, Yudong Tao, Shu Ching Chen, Mei Ling Shyu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

During disaster events, emergency response teams need to draw up the response plan at the earliest possible stage. Social media platforms contain rich information which could help to assess the current situation. In this paper, a novel multi-Task multimodal deep learning framework with automatic loss weighting is proposed. Our framework is able to capture the correlation among different concepts and data modalities. The proposed automatic loss weighting method can prevent the tedious manual weight tuning process and improve the model performance. Extensive experiments on a large-scale multimodal disaster dataset from Twitter are conducted to identify post-disaster humanitarian category and infrastructure damage level. The results show that by learning the shared latent space of multiple tasks with loss weighting, our model can outperform all single tasks.

Original languageEnglish (US)
Title of host publicationProceedings - 3rd International Conference on Multimedia Information Processing and Retrieval, MIPR 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages209-212
Number of pages4
ISBN (Electronic)9781728142722
DOIs
StatePublished - Aug 2020
Event3rd International Conference on Multimedia Information Processing and Retrieval, MIPR 2020 - Shenzhen, Guangdong, China
Duration: Aug 6 2020Aug 8 2020

Publication series

NameProceedings - 3rd International Conference on Multimedia Information Processing and Retrieval, MIPR 2020

Conference

Conference3rd International Conference on Multimedia Information Processing and Retrieval, MIPR 2020
Country/TerritoryChina
CityShenzhen, Guangdong
Period8/6/208/8/20

Keywords

  • disaster information management
  • multi-Task
  • multimodal learning

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Signal Processing
  • Media Technology
  • Modeling and Simulation
  • Library and Information Sciences

Fingerprint

Dive into the research topics of 'Multi-Task Multimodal Learning for Disaster Situation Assessment'. Together they form a unique fingerprint.

Cite this