A major challenge for fMRI studies is to use realistic stimuli, which is relevant for the understanding of multisensory interactions in the real world. Computational models [Itti et al. 1998; Kayser et al. 2005] have been used to track brain activity during viewing of complex audio-visual stimuli [Bartels et al. 2007; Bordier et al. 2013]. We extended this approach to investigate activity associated with high-order spatial aspects in both vision and audition. We utilized a 3D-surround movie that included visual disparity cues and multiple sound sources (centre, front left/right, back left/right). For each visual frame, we computed a disparity map [Liu et al. 2011] and indexed absolute disparity (sum over the entire map) and gradient disparity (local contrast, Bordier et al. 2013). For audition, we indexed sound-spatiality by computing the correlation between each of the 5 external channels and a sixth channel that was delivered over headphones during fMRI (this contained primarily the “centre” sound). We also indexed the sound intensity contrast [Bordier et al. 2013] to control for mere intensity changes over time. These indexes were used to fit the BOLD signal in 16 subjects, who watched the 3D-surround movie during fMRI. The results showed a dissociation between absolute disparity (PPC and V3A) and gradient disparity (V6, STS and IFG). The auditory-spatiality index correlated with activity in auditory cortex, even after accounting for changes of auditory-intensity. We conclude that the combination of computational models and fMRI enables studying brain activity associated with complex spatial signals in naturalistic multisensory environments.

Computational models; visual disparity; multiple sound sources