visual speech affects discrimination of syllables in the auditory cortex: an MMN study

Kislyuk Daniil, Cognitive Technology Group, Laboratory of Computational Engineering, Helsinki University of Technology

Abstract
Infrequent changes in the visual component of audiovisual speech stimuli, resulting in the “McGurk effect”, elicit a mismatch response (MMN) in the auditory cortex even when the auditory stream remains constant (Sams et al., 1991; Möttönen et al., 2002; Colin et al., 2002). This result has been interpreted to suggest that visual information modifies processing of auditory information in the auditory cortex. However, this MMN may arise due to the change in the visual stimulus per se.
In the present EEG experiment, we avoided this possibility by keeping the visual sequence constant. An odd-ball sequence of auditory stimuli consisting of frequent /va/ syllables (standards) and infrequent /ba/ syllables (deviants) was presented to nine subjects. Deviant stimuli in the unisensory acoustic stimulus sequence elicited a typical MMN, indicating stimulus discrimination. When the acoustic stimuli were dubbed onto a video of mouth constantly articulating /va/, the deviant acoustic /ba/ was perceived as /va/ due to the McGurk effect and was perceptually indistinguishable from the standards. Now acoustic deviants did not elicit any mismatch response, indicating that auditory cortex did not discriminate between the acoustic stimuli anymore.
Our finding suggests that processing visual information can change neural representations underlying auditory discrimination similarly as occurs when the properties of the acoustic stimulus are changed.

Not available

Back to Abstract