Integrating Image and Textual Information in Human-Robot Interactions for Children with Autism Spectrum Disorder (ASD)

Xue Yang, Mei-Ling Shyu, Han Qi Yu, Shi ming Sun, Nian Sheng Yin, Wei Chen

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

Talking and literary reading are important activities for children, especially for children with Autism Spectrum Disorder (ASD). We try to integrate the activities with NAO robots to excite their communication willingness. Making book choices according to the conversation is a task for NAO robots. In this paper, a novel multi-modal picture book recommendation framework that combines textual information and image information to calculate the similarity between the picture books and the conversation topics is proposed and evaluated using a testing dataset. In the proposed framework, an image neighbor discovery method to get more relative terms and an NDK (Near-Duplicated Keyframes) friend detection method to get more relative NDKs are proposed. Both correspondence analysis (CA) and multiple correspondence analysis (MCA) algorithms are utilized. Finally, the booklist generated from the experiment is evaluated by 6 performance indicators and the experimental results demonstrate that our proposed framework achieves satisfactory and promising performance. With the help of the proposed recommendation framework, an autistic child can talk to the NAO robot in a relaxed and enjoyable environment. Please note that the proposed framework is not evaluated for its performance with the ASD children but for its performance at recommending books based on visual and textual features. Therefore, no tests were performed with either professionals nor diagnosed individuals.

Original languageEnglish (US)
JournalIEEE Transactions on Multimedia
DOIs
StateAccepted/In press - Aug 16 2018

Fingerprint

Human robot interaction
Robots
Communication
Testing
Experiments

Keywords

  • Autism Spectrum Disorder (ASD)
  • human-robot interaction
  • integration
  • multi-modality
  • multiple correspondence analysis (MCA)
  • nearduplicated keyframe (NDK)
  • picture book recommendation
  • Robot

ASJC Scopus subject areas

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this

Integrating Image and Textual Information in Human-Robot Interactions for Children with Autism Spectrum Disorder (ASD). / Yang, Xue; Shyu, Mei-Ling; Yu, Han Qi; Sun, Shi ming; Yin, Nian Sheng; Chen, Wei.

In: IEEE Transactions on Multimedia, 16.08.2018.

Research output: Contribution to journalArticle

@article{a8648e5d8b7c440383a0da7e2d1f66d2,
title = "Integrating Image and Textual Information in Human-Robot Interactions for Children with Autism Spectrum Disorder (ASD)",
abstract = "Talking and literary reading are important activities for children, especially for children with Autism Spectrum Disorder (ASD). We try to integrate the activities with NAO robots to excite their communication willingness. Making book choices according to the conversation is a task for NAO robots. In this paper, a novel multi-modal picture book recommendation framework that combines textual information and image information to calculate the similarity between the picture books and the conversation topics is proposed and evaluated using a testing dataset. In the proposed framework, an image neighbor discovery method to get more relative terms and an NDK (Near-Duplicated Keyframes) friend detection method to get more relative NDKs are proposed. Both correspondence analysis (CA) and multiple correspondence analysis (MCA) algorithms are utilized. Finally, the booklist generated from the experiment is evaluated by 6 performance indicators and the experimental results demonstrate that our proposed framework achieves satisfactory and promising performance. With the help of the proposed recommendation framework, an autistic child can talk to the NAO robot in a relaxed and enjoyable environment. Please note that the proposed framework is not evaluated for its performance with the ASD children but for its performance at recommending books based on visual and textual features. Therefore, no tests were performed with either professionals nor diagnosed individuals.",
keywords = "Autism Spectrum Disorder (ASD), human-robot interaction, integration, multi-modality, multiple correspondence analysis (MCA), nearduplicated keyframe (NDK), picture book recommendation, Robot",
author = "Xue Yang and Mei-Ling Shyu and Yu, {Han Qi} and Sun, {Shi ming} and Yin, {Nian Sheng} and Wei Chen",
year = "2018",
month = "8",
day = "16",
doi = "10.1109/TMM.2018.2865828",
language = "English (US)",
journal = "IEEE Transactions on Multimedia",
issn = "1520-9210",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - JOUR

T1 - Integrating Image and Textual Information in Human-Robot Interactions for Children with Autism Spectrum Disorder (ASD)

AU - Yang, Xue

AU - Shyu, Mei-Ling

AU - Yu, Han Qi

AU - Sun, Shi ming

AU - Yin, Nian Sheng

AU - Chen, Wei

PY - 2018/8/16

Y1 - 2018/8/16

N2 - Talking and literary reading are important activities for children, especially for children with Autism Spectrum Disorder (ASD). We try to integrate the activities with NAO robots to excite their communication willingness. Making book choices according to the conversation is a task for NAO robots. In this paper, a novel multi-modal picture book recommendation framework that combines textual information and image information to calculate the similarity between the picture books and the conversation topics is proposed and evaluated using a testing dataset. In the proposed framework, an image neighbor discovery method to get more relative terms and an NDK (Near-Duplicated Keyframes) friend detection method to get more relative NDKs are proposed. Both correspondence analysis (CA) and multiple correspondence analysis (MCA) algorithms are utilized. Finally, the booklist generated from the experiment is evaluated by 6 performance indicators and the experimental results demonstrate that our proposed framework achieves satisfactory and promising performance. With the help of the proposed recommendation framework, an autistic child can talk to the NAO robot in a relaxed and enjoyable environment. Please note that the proposed framework is not evaluated for its performance with the ASD children but for its performance at recommending books based on visual and textual features. Therefore, no tests were performed with either professionals nor diagnosed individuals.

AB - Talking and literary reading are important activities for children, especially for children with Autism Spectrum Disorder (ASD). We try to integrate the activities with NAO robots to excite their communication willingness. Making book choices according to the conversation is a task for NAO robots. In this paper, a novel multi-modal picture book recommendation framework that combines textual information and image information to calculate the similarity between the picture books and the conversation topics is proposed and evaluated using a testing dataset. In the proposed framework, an image neighbor discovery method to get more relative terms and an NDK (Near-Duplicated Keyframes) friend detection method to get more relative NDKs are proposed. Both correspondence analysis (CA) and multiple correspondence analysis (MCA) algorithms are utilized. Finally, the booklist generated from the experiment is evaluated by 6 performance indicators and the experimental results demonstrate that our proposed framework achieves satisfactory and promising performance. With the help of the proposed recommendation framework, an autistic child can talk to the NAO robot in a relaxed and enjoyable environment. Please note that the proposed framework is not evaluated for its performance with the ASD children but for its performance at recommending books based on visual and textual features. Therefore, no tests were performed with either professionals nor diagnosed individuals.

KW - Autism Spectrum Disorder (ASD)

KW - human-robot interaction

KW - integration

KW - multi-modality

KW - multiple correspondence analysis (MCA)

KW - nearduplicated keyframe (NDK)

KW - picture book recommendation

KW - Robot

UR - http://www.scopus.com/inward/record.url?scp=85051824266&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85051824266&partnerID=8YFLogxK

U2 - 10.1109/TMM.2018.2865828

DO - 10.1109/TMM.2018.2865828

M3 - Article

AN - SCOPUS:85051824266

JO - IEEE Transactions on Multimedia

JF - IEEE Transactions on Multimedia

SN - 1520-9210

ER -