Multisensory illusions have often been used to study the principles of multisensory binding processes. For example, when a central sound is paired with two lateralized visual stimuli, the sound is perceived at the location of the matching visual stimulus. This effect is well known as the ventriloquist illusion and is thought to indicate the dominance of vision for sensory localization. The present study had two goals: (1) to test whether affective expression is capable to act similarly as temporal synchrony as supramodal binding feature; (2) to establish a paradigm that allowed measuring the ventriloquist illusion response-bias free.
Two movies of a speaker were presented one in the left and in the right hemifield; the sound was presented form a central location. The actors spoke bi-syllabic pseudo-words. In the experimental condition, one face matched the voice while the other face did not match the spoken word. The match consisted either of speaking the same vs. a different word or of speaking the same word with the same or a different affective expression. The participants had to decide whether the word was spoken with a loud or a low voice. If a ventriloquist effect occurred based on temporal or affective binding, respectively, we expected response times to be lower for loud-low decisions that required a response with the ipsilateral hand, thus we predicted a Simon effect to occur. We ran a number of control conditions including a real lateralized presentation of the voice. Results showed a Simon effect both for the temporal and affective ventriloquist conditions although smaller than for the real lateralized control conditions.
Thus, we established a paradigm that allows accessing the ventriloquist effect response bias free. Moreover, we demonstrated automatic multisensory binding based on affective expression.