Audio-visual integration of emotional information
Penny Bergman, Daniel Västfjäll, Ana Tajadura-Jiménez

Last modified: 2011-09-02

Abstract


Emotions are central to our perception of the environment surrounding us (Berlyne, 1971). An important aspect in the emotional response to a sound is dependent on the meaning of the sound, i.e. it is not the physical parameter per se that determines our emotional response to the sound but rather the source of the sound (Genell, 2008), and the relevance it has to the self (Tajadura-Jiménez et al., 2010). When exposed to sound together with visual information the information from both modalities is integrated, altering the perception of each modality, in order to generate a coherent experience. In emotional information this integration is rapid and without requirements of attentional processes (De Gelder, 1999). The present experiment investigates perception of pink noise in two visual settings in a within-subjects design. 19 participants rated the same sound twice in terms of pleasantness and arousal in either a pleasant or an unpleasant visual setting. The results showed that pleasantness of the sound decreased in the negative visual setting, thus suggesting an audio-visual integration, where the affective information in the visual modality is translated to the auditory modality when information-markers are lacking in it. The results are discussed in relation to theories of emotion perception.

References


Berlyne, D.E. (1971). Aesthetics and psychobiology. New York: Appleton Century Crofts.

De Gelder, B. (1999). Seeing cries and hearing smiles: Crossmodal perception of emotional expressions. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events (pp. 425-438). Elsevier Science.

Genell, A. (2008). Perception of sound and vibration in heavy trucks. PhD thesis, Chalmers University of Technology, Gothenburg, Sweden.

Tajadura-Jiménez, A., Väljamäe, A., Asutay, E., & Västfjäll, D. (2010). Embodied auditory perception: The emotional impact of approaching and receding sound sources. Emotion, 10, 216-229.

Conference System by Open Conference Systems & MohSho Interactive