Where’s Wally? Audio-visual mismatch directs ocular saccades in sensory substitution
Thomas David Wright, Jamie Ward, Sarah Simonon, Aaron Margolis

Date: 2012-06-19 01:30 PM – 03:00 PM
Last modified: 2012-04-27

Abstract


Sensory substitution is the representation of information from one sensory modality (e.g. vision) within another modality (e.g. audition). We used a visual-to-auditory sensory substitution device (SSD) to explore the effect of incongruous (true-)visual and substituted-visual signals on visual attention. In our multisensory sensory substitution paradigm, both visual and sonified-visual information were presented. By making small alterations to the sonified image, but not the seen image, we introduced audio-visual mismatch. The alterations consisted of the addition of a small image (for instance, the Wally character from the “Where’s Wally?” books) within the original image. Participants were asked to listen to the sonified image and identify which quadrant contained the alteration. Monitoring eye movements revealed the effect of the audio-visual mismatch on covert visual attention. We found that participants consistently fixated more, and dwelled for longer, in the quadrant corresponding to the location (in the sonified image) of the target. This effect was not contingent on the participant reporting the location of the target correctly, which indicates a low-level interaction between an auditory stream and visual attention. We propose that this suggests a shared visual workspace that is accessible by visual sources other than the eyes. If this is indeed the case, it would support the development of other, more esoteric, forms of sensory substitution. These could include an expanded field of view (e.g. rear-view cameras), overlaid visual information (e.g. thermal imaging) or restoration of partial visual field loss (e.g. hemianopsia).

Conference System by Open Conference Systems & MohSho Interactive