Integrating image and textual information in human–robot interactions for children with autism spectrum disorder

Xue Yang, Mei Ling Shyu, Han Qi Yu, Shi Ming Sun, Nian Sheng Yin, Wei Chen

Research output: Contribution to journalArticlepeer-review

7 Scopus citations


Talking and literary reading are important activities for children, especially for children with autism spectrum disorder (ASD). We try to integrate the activities with NAO robots to excite their communication willingness. In this paper, a novel multimodal picture book recommendation framework that combines textual information and image information to calculate the similarity between the picture books and the conversation topics is proposed and evaluated using a testing dataset. In the proposed framework, an image neighbor discovery method to get more relative terms and an near-duplicated keyframes (NDK) friend detection method to get more relative NDKs are proposed. Finally, the booklist generated from the experiment is evaluated by six performance indicators and the experimental results demonstrate that our proposed framework achieves satisfactory and promising performance. With the help of the proposed recommendation framework, an autistic child can talk to the NAO robot in a relaxed and enjoyable environment. Please note that the proposed framework is not evaluated for its performance with the ASD children but for its performance at recommending books based on visual and textual features. Therefore, no tests were performed with either professionals nor diagnosed individuals.

Original languageEnglish (US)
Article number8438947
Pages (from-to)746-759
Number of pages14
JournalIEEE Transactions on Multimedia
Issue number3
StatePublished - Mar 2019


  • Autism spectrum disorder (ASD)
  • Human-robot interaction
  • Integration
  • Multi-modality
  • Multiple correspondence analysis (MCA)
  • Near-duplicated keyframe (NDK)
  • Picture book recommendation
  • Robot

ASJC Scopus subject areas

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Integrating image and textual information in human–robot interactions for children with autism spectrum disorder'. Together they form a unique fingerprint.

Cite this