School of Informatics and Computing Menu

Chen Yu, Ph.D., Understanding Micro-Level Multimodal Behaviors in Child-Parent and Human-Robot Interactions



Interacting embodied agents, be they groups of adults engaged in a coordinated task, a mother teaching a child, or autonomous robots acting in an environment, must seamlessly coordinate their actions to achieve a collaborative goal. The pursuit of a shared goal requires mutual recognition of the goal, appropriate sequencing and coordination of each agent’s behavior with others, and making predictions from and about the likely behavior of others. In this talk, Dr. Yu will first present a set of studies using advanced sensing equipment and data mining techniques to collect and analyze dynamic multi-coupling sensory data in child-parent interactions. The results show that inter-agent coordination at least at an elementary level depends crucially on external (and observable) behaviors by the participants where the real-time behavior of one participant organizes the actions of the other–behaviors such as eye movements, head turns, and hand gestures–and those micro-level behaviors are predictive to successful word learning through child-parent free-play interaction. Next, I will present a multimodal human-robot interaction experiment using a Nao robot as an embodied model. A gaze-contingent design was implemented in which we systematically manipulated the robot’s gaze to the human partner’s face in real time and then analyzed the human’s gaze behavior as a response to the robot’s gaze behavior. The results show participants demonstrated more coordinated and synchronized multimodal behaviors between speech and gaze when more eye contact was successfully established and maintained.


Chen Yu is an Associate Professor in the Department of Psychological and Brain Sciences, Cognitive Science and Computer Science at Indiana University. He received his Ph.D. in computer science from University of Rochester in 2004 and joined Indiana University since then. Dr. Yu has received David Marr best paper award from Cognitive Science Society and an outstanding junior faculty award from Indiana University at Bloomington. His research is supported by NIH, NSF and AFOSR. Dr. Yu has published papers in the fields of cognitive science, developmental psychology, and computer science. He has served as panelists in several NSF panels and NIH study sections.