Utility of gestural cues in indexing semantic miscommunication

Masashi Inoue, Mitsunori Ogihara, Ryoko Hanada, Nobuhiro Furuyama

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

In multimedia data analysis, automated indexing of conversational video is an emerging topic. One challenging problem in this topic is the recognition of higher-level concepts, such as miscommunications in conversations. While detecting miscommunications is generally easy for speakers as well as observers, it is not currently understood which cues contribute to their detection and to what extent. To make use of the knowledge on gestural cues in multimedia systems, the applicability of machine learning is investigated as a means of detecting miscommunication from gestural patterns observed in psychotherapeutic face-to-face conversations. Various features are taken from gesture data, and both simple and complex classifiers are constructed using these features. Both short-term and long-term effects are tested using different time window sizes. Also, two types of gestures, communicative and non-communicative, are considered. The experimental results suggest that there is no single gestural feature that can explain the occurrence of semantic miscommunication. Another interesting finding is that gestural cues correlate more with long-term gestural patterns than with short-term ones.

Original languageEnglish (US)
Title of host publication2010 5th International Conference on Future Information Technology, FutureTech 2010 - Proceedings
DOIs
StatePublished - 2010
Event5th International Conference on Future Information Technology, FutureTech 2010 - Busan, Korea, Republic of
Duration: May 20 2010May 24 2010

Other

Other5th International Conference on Future Information Technology, FutureTech 2010
CountryKorea, Republic of
CityBusan
Period5/20/105/24/10

Fingerprint

Multimedia systems
Learning systems
Classifiers
Semantics

Keywords

  • Face-toface
  • Gesture
  • Psychotherapy
  • Semantic indexing

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems

Cite this

Inoue, M., Ogihara, M., Hanada, R., & Furuyama, N. (2010). Utility of gestural cues in indexing semantic miscommunication. In 2010 5th International Conference on Future Information Technology, FutureTech 2010 - Proceedings [5482653] https://doi.org/10.1109/FUTURETECH.2010.5482653

Utility of gestural cues in indexing semantic miscommunication. / Inoue, Masashi; Ogihara, Mitsunori; Hanada, Ryoko; Furuyama, Nobuhiro.

2010 5th International Conference on Future Information Technology, FutureTech 2010 - Proceedings. 2010. 5482653.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Inoue, M, Ogihara, M, Hanada, R & Furuyama, N 2010, Utility of gestural cues in indexing semantic miscommunication. in 2010 5th International Conference on Future Information Technology, FutureTech 2010 - Proceedings., 5482653, 5th International Conference on Future Information Technology, FutureTech 2010, Busan, Korea, Republic of, 5/20/10. https://doi.org/10.1109/FUTURETECH.2010.5482653
Inoue M, Ogihara M, Hanada R, Furuyama N. Utility of gestural cues in indexing semantic miscommunication. In 2010 5th International Conference on Future Information Technology, FutureTech 2010 - Proceedings. 2010. 5482653 https://doi.org/10.1109/FUTURETECH.2010.5482653
Inoue, Masashi ; Ogihara, Mitsunori ; Hanada, Ryoko ; Furuyama, Nobuhiro. / Utility of gestural cues in indexing semantic miscommunication. 2010 5th International Conference on Future Information Technology, FutureTech 2010 - Proceedings. 2010.
@inproceedings{9859b9c317f2480382b1ff7db5fd8912,
title = "Utility of gestural cues in indexing semantic miscommunication",
abstract = "In multimedia data analysis, automated indexing of conversational video is an emerging topic. One challenging problem in this topic is the recognition of higher-level concepts, such as miscommunications in conversations. While detecting miscommunications is generally easy for speakers as well as observers, it is not currently understood which cues contribute to their detection and to what extent. To make use of the knowledge on gestural cues in multimedia systems, the applicability of machine learning is investigated as a means of detecting miscommunication from gestural patterns observed in psychotherapeutic face-to-face conversations. Various features are taken from gesture data, and both simple and complex classifiers are constructed using these features. Both short-term and long-term effects are tested using different time window sizes. Also, two types of gestures, communicative and non-communicative, are considered. The experimental results suggest that there is no single gestural feature that can explain the occurrence of semantic miscommunication. Another interesting finding is that gestural cues correlate more with long-term gestural patterns than with short-term ones.",
keywords = "Face-toface, Gesture, Psychotherapy, Semantic indexing",
author = "Masashi Inoue and Mitsunori Ogihara and Ryoko Hanada and Nobuhiro Furuyama",
year = "2010",
doi = "10.1109/FUTURETECH.2010.5482653",
language = "English (US)",
isbn = "9781424469505",
booktitle = "2010 5th International Conference on Future Information Technology, FutureTech 2010 - Proceedings",

}

TY - GEN

T1 - Utility of gestural cues in indexing semantic miscommunication

AU - Inoue, Masashi

AU - Ogihara, Mitsunori

AU - Hanada, Ryoko

AU - Furuyama, Nobuhiro

PY - 2010

Y1 - 2010

N2 - In multimedia data analysis, automated indexing of conversational video is an emerging topic. One challenging problem in this topic is the recognition of higher-level concepts, such as miscommunications in conversations. While detecting miscommunications is generally easy for speakers as well as observers, it is not currently understood which cues contribute to their detection and to what extent. To make use of the knowledge on gestural cues in multimedia systems, the applicability of machine learning is investigated as a means of detecting miscommunication from gestural patterns observed in psychotherapeutic face-to-face conversations. Various features are taken from gesture data, and both simple and complex classifiers are constructed using these features. Both short-term and long-term effects are tested using different time window sizes. Also, two types of gestures, communicative and non-communicative, are considered. The experimental results suggest that there is no single gestural feature that can explain the occurrence of semantic miscommunication. Another interesting finding is that gestural cues correlate more with long-term gestural patterns than with short-term ones.

AB - In multimedia data analysis, automated indexing of conversational video is an emerging topic. One challenging problem in this topic is the recognition of higher-level concepts, such as miscommunications in conversations. While detecting miscommunications is generally easy for speakers as well as observers, it is not currently understood which cues contribute to their detection and to what extent. To make use of the knowledge on gestural cues in multimedia systems, the applicability of machine learning is investigated as a means of detecting miscommunication from gestural patterns observed in psychotherapeutic face-to-face conversations. Various features are taken from gesture data, and both simple and complex classifiers are constructed using these features. Both short-term and long-term effects are tested using different time window sizes. Also, two types of gestures, communicative and non-communicative, are considered. The experimental results suggest that there is no single gestural feature that can explain the occurrence of semantic miscommunication. Another interesting finding is that gestural cues correlate more with long-term gestural patterns than with short-term ones.

KW - Face-toface

KW - Gesture

KW - Psychotherapy

KW - Semantic indexing

UR - http://www.scopus.com/inward/record.url?scp=77954400664&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77954400664&partnerID=8YFLogxK

U2 - 10.1109/FUTURETECH.2010.5482653

DO - 10.1109/FUTURETECH.2010.5482653

M3 - Conference contribution

SN - 9781424469505

BT - 2010 5th International Conference on Future Information Technology, FutureTech 2010 - Proceedings

ER -