Cross-modal perception of complex visual-haptic stimuli
Single Paper Presentation

Ian Oakley
Media Lab Europe

Sile O'Modhrain
Media Lab Europe

     Abstract ID Number: 107
     Full text: Not available
     Last modified: May 20, 2003

Abstract
Our visual and haptic perceptual systems are inextricably intertwined. Research into cross-modal object recognition (Newell et al., 2001) has highlighted this fact, and studies examining the influence of visual distracters on a tactile localisation task (Pavani et al., 2000) serve to reinforce this with the assertion that, in certain situations, vision can capture touch. In other words, a distracting visual stimuli can delay, and even make erroneous, the perception of the location of a tactile stimulus on the skin. Here we extend this work by examining more complex interactions between visual and haptic cues. We are concerned with how the human perceptual system can integrate a complex, dynamic visual scene and a kinaesthetic cue, in the form of a force, that relates to some aspect of this scene. Essentially, we address the question of whether or not kinaesthetic information can cause a particular visual stimuli to “pop-out” of a complex scene. In our current experimental design, we are investigating the graphical display of a number of bouncing balls, and the kinaesthetic display (using a PHANToM haptic interface device) of the velocity of one of these balls, or of an unrelated velocity profile. The subject’s task is to determine whether or not the kinaesthetic cue relates to one of the displayed balls. We are interested in the how the completion time and accuracy rate of this task changes as the number of visually presented balls increases; will the haptic cues cause the associated visual stimuli to pop-out of the scene?


References
Newell F.N., Ernst M.O., Tjan B.S., and Bulthoff H.H. (2001). Viewpoint dependence in visual and haptic object recognition. Psychological Science, 12 (1): 37-42.

Pavani, F., Spence, C. and Driver, J. (2000). Visual Capture of Touch: Out-of-the-body Experiences With Rubber Gloves. Psychological Science, 11(5): 353-359