Multi-modal attention and event binding in humanoid robots using a sensory ego-sphere

Abstract

The sensory abilities of humanoid robots are progressing every year with the development of advanced and complex sensors. At the same time, humanoid robots are learning to coordinate sensory events in their environments with the actions that they produce. This dissertation presents results of mechanisms developed to allow a humanoid robot to discover important sensory events, both internal and external, and to determine which of those events requires attention. The mechanisms are implemented on the Sensory Ego-Sphere (SES), a spatio-temporal data structure used by the robot as an interface between sensing and cognition. The mechanisms presented are event binding and attention. Attention gives a robot the ability to focus on areas of its environment that are salient, that contain events related to the robot’s current tasks or events that are unexpected and sudden. It also enables the robot to disregard irrelevant areas, that contain events unrelated to current tasks or events that are habitual. Event binding selects all co-occurring events near the focus of attention that originated from the same source. The architecture that uses these capabilities and designs for attention and event binding mechanisms are presented. Salience of events is determined by incidence of the event on the SES, relevance of the event to current tasks and habituation of the event over time. Therefore, salience spikes around events that occur in the same area around the same time. The area with the highest salience is selected as the focus of attention. Event binding examines the spatial and temporal features of events in the focal area and tags those events as co-occurring. Experiments were performed on ISAC, Vanderbilt University’s humanoid and Robonaut, the NASA/DARPA humanoid. Results from these experiments are presented and discussed. The results show that with the Sensory Ego-Sphere, a humanoid can detect the most salient areas of its environment and bind together those events that originated from a single source.