Amodal Multimodal Integration

Marc O. Ernst, Massimiliano Di Luca, Wendy Adams
Symposium Talk
Last modified: 2008-05-13

Abstract


Recently it has been shown that congruent visual and haptic signals
are integrated in a statistically optimal fashion. Spatial separation
between the signals can preclude this integration. Here we investigate
whether optimal integration occurs between an amodally completed
visual stimulus and its haptic counterpart. Thus, we ask whether
integration occurs despite the sensory information not being derived
from the same spatial location. This may indicate that subjects
inferred that the visually specified parts of the stimulus and the
haptic information have a common cause and thus should be integrated.

The visual stimulus was a disparity-defined bar that was partially
occluded (amodal completion condition). The bar could also be touched
behind the occluder using two fingers. Subjects’ task was to
discriminate the size of two successively presented bars using a 2-IFC
paradigm, where one interval contained conflicting haptic and visual
information. Performance in the amodal completion condition was not
different from a condition in which the occluder was removed (visual-
haptic condition). Both conditions were consistent with an optimal
integration strategy.

More interestingly, integration deviated from optimality when we
introduced a slight modification to the visual stimulus – small gaps
between the bar and the occluder (gap condition). This manipulation
interfered with the amodal completion process and consequently
subjects relied almost completely on the haptic information for
discriminating the size of the bars.

These findings suggest that visual and haptic information can be
combined optimally even when visual information is not directly
specified by sensory information, but results from amodal completion.
In conclusion, it seems that the perceptual system determines when to
combine visual and haptic information based on the likelihood the
signals have of belonging to the same object (i.e. if there is a
causal relationship between the signals) and not only on signal co-
location.

Conference System by Open Conference Systems & MohSho Interactive Multimedia