Towards Building Rapport with a Human Support Robot

Katarzyna Pasternak, Zishi Wu, Ubbo Visser, Christine Lisetti

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Human support robots (mobile robots able to perform useful domestic manipulative tasks) might be better accepted by people if they can communicate in ways they naturally understand: e.g. speech, but also facial expressions, postures, among others. Subtle (unconscious) mirroring of nonverbal cues during conversations promotes rapport building, essential for good communication. We investigate whether, as in human-human communication, the ability of a robot to mirror its user’s head movements and facial expressions in real time can improve the user’s experience with it. We describe the technical integration of a Toyota Human Support Robot (HSR) with a facially expressive 3D embodied conversational agent (ECA) (named ECA-HSR). The HSR and the ECA are aware of the user’s head movements and facial emotions, and can mirror them, in real time. We then discuss a user study we designed in which participants interacted with ECA-HSR in a simple social dialog task with three conditions: mirroring of user’s head movements, mirroring of user’s facial emotions, and mirroring of both user’s head movements and facial emotions. Our results suggest that interacting with an ECA-HSR that mirrors both the user’s head movements and the facial expressions is preferred over the other conditions. Among other insights, the study revealed that the accuracy of open source, real-time recognition of facial expressions of emotion needs improvement for the best user’s acceptance.

Original languageEnglish (US)
Title of host publicationRoboCup 2021
Subtitle of host publicationRobot World Cup XXIV
EditorsMaya Cakmak, Oliver Obst
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages12
ISBN (Print)9783030986810
StatePublished - 2022
Externally publishedYes
Event24th RoboCup International Symposium, RoboCup 2021 - Virtual, Online
Duration: Jun 22 2021Jun 28 2021

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13132 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference24th RoboCup International Symposium, RoboCup 2021
CityVirtual, Online


  • 3d embodied conversational agents
  • Autonomous support robots
  • Human robot interaction
  • Nonverbal communication

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'Towards Building Rapport with a Human Support Robot'. Together they form a unique fingerprint.

Cite this