Introduction: Synchrony is a powerful cue for driving multisensory integration of dynamic stimuli and multisensory integration increases perceptual reliability. Thus ambiguous, dynamic, visual targets should be more accurately perceived when accompanied by concurrent auditory stimuli. For example, when a creature moving through the undergrowth is obscured by both intervening foliage & conditions of poor illumination, it may be more readily identified when the sound of each footfall is audible - heralding motion of figure against background and foreground. This fMRI study was designed to identify the neural correlates of synchrony-induced multisensory integration during shape and motion discrimination.
Methods: 16 subjects participated in this fMRI study (Siemens Allegra 3 T scanner, GE-EPI, TE=30, 38 axial slices, TR=3.08s). The 2x2 factorial design manipulated audiovisual synchrony (synchronous vs. asynchronous) and task (motion vs. shape discrimination). In a visual selective attention paradigm, subjects discriminated the shape or rotational motion of a dot array obscured by a rapidly-moving snow field. Each 2.5s trial consisted of 5 visual events that occurred at unpredictable intervals to eliminate anticipatory effects associated with regular stimulus timings. In 50% of the trials, each visual event was accompanied by a simultaneous 50ms “click� sound (synchronous condition), whilst in the other 50% the auditory and visual events were offset by at least 120ms (asynchronous condition). Visual events were discrete rotations of one of 32 different symmetrical, concentrically distributed, dot arrays. Synchronous and asynchronous trials were presented in randomised order and task order was counterbalanced within and across subjects. To allow for a random-effects analysis (SPM5), contrast images for each subject were entered into second level one-sample t-tests. We tested for the main effects of synchrony and task and their interactions. Effects are reported at p<0.05 whole brain corrected.
Behavioural Results: Subject performance was significantly more accurate under conditions of visual-auditory synchrony than asynchrony, whilst there was no significant difference in performance between the shape and motion tasks. The interactions indicate a significant performance improvement during synchronous trials for the motion task, but not the shape task.
fMRI Results: In bilateral posterior occipital cortices audiovisual synchrony amplified the BOLD response, irrespective of task. In contrast, in lateral occipital cortex (LOC) and posterior superior temporal (pSTG) / supramarginal (SMG) gyri, synchrony-induced activation increases were task dependent: in LOC synchrony effects were selective for shape discrimination, whilst in pSTG/SMG they were selective for motion discrimination.
Discussion/conclusion: Our results suggest that, under noisy visual conditions, visual perception can be enhanced by concurrent acoustic stimulation in the following way: Audiovisual coincidence may enhance saliency of visual stimuli by amplification of visual responses in occipital cortex and thus lead to improved figure-ground segmentation. These are then further amplified in higher order LOC and pSTG/SMG in a task-selective fashion.