Effects of within-modal congruency, cross-modal congruency and temporal asynchrony on the perception of perceived audio-visual distance
Date: 2012-06-21 01:30 PM – 03:00 PM
Last modified: 2012-04-27
Abstract
The factors we use to determine whether information from separate modalities should be assigned to the same source include task demands, the spatial and temporal coincidence of the composite signals, and, whether the signals are congruent with one another. In a series of experiments, we examined how temporal asynchrony and congruency interact in a competitive binding situation. Across a series of experiments, participants assigned a temporally roving auditory stimulus to competing primary or secondary visual anchors (VAV), or, a temporally roving visual stimulus to competing primary or secondary auditory anchors (AVA), based on causality. Congruency was defined in terms of simulated distance both within- and between-modalities (visual: small, auditory: quiet = far; visual: large, auditory: loud = near). Strong temporal effects were revealed, with differences between VAV and AVA conditions reflecting natural auditory lag tolerance for binding. During VAV conditions, binding was influenced only by visual congruency. During AVA conditions, binding was influenced by audio-visual congruency. These differences did not seem to be due to the relative discriminability between visual and auditory magnitude. The data reiterate the dominance of audition in the time domain (showing stronger temporal effects), the dominance of vision in the spatial domain (showing stronger congruency effects), and, the assistance of domain-inappropriate modalities by domain-appropriate modalities. A special case of congruency in terms of visual looming will also be discussed, along with the potential alerting properties of high magnitude stimuli.
References
Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14 (3), 257-262.
Alais, D., & Carlile, S. (2005). Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. Proceedings of the National Academy of Sciences of the United States of America, 102 (6), 2244-2247.
Arnold, D. H., Johnston, A., & Nishida, S. (2005). Timing sight and sound. Vision Research, 45, 1275-1284.
Bermant, R. I., & Welch, R. B. (1976). Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Perceptual and Motor Skills, 43, 487-493.
Bien, N., ten Oever, S., Goebel, R., & Sack, A. T. (2012). The sound of size: crossmodal binding in pitch-size synesthesia: a combined TMS, EEG and psychophysics study. NeuroImage, 59 (1), 663-672.
Burr, D., Banks, M. S., & Morrone, M. C. (2009). Auditory dominance over vision in the perception of interval duration. Experimental Brain Research,198 (1), 49-57.
Calvert, G. A., Spence, C., & Stein, B. E. (2004). The handbook of multisensory processing. Cambridge, MA: MIT Press.
Cappe, C., Thut, G., Romei, V., & Murray, M. M. (2009). Selective integration of auditory-visual looming cues by humans. Neuropsychologia, 47 (4), 1045-1052.
Cohen, J., MacWhinney, B., Flatt, M. & Provost, J. (1993). PsyScope: An interactive graphic system for designing and controlling experiments in the psychology laboratory using Macintosh computers. Behavior Research Methods, Instruments, and Computers, 25, 257-271.
Dixon, N. F., & Spitz, L. (1980). The detection of auditory visual desynchrony. Perception, 9, 719-721.
Engel, G. R., & Dougherty, W. G. (1971). Visual-auditory distance constancy. Nature, 234, 308.
Evans, K. K., & Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features. Journal of Vision, 10 (1), 1-12.
Fujisaki, W., Shimojo, S., Kashino, S. & Nishida, S. (2004). Recalibration of audio-visual simultaneity. Nature Neuroscience, 7, 773-778.
Fujisaki, W., Koene, A., Arnold, D., Johnston, A., & Nishida, S. (2006). Visual search for a target changing in synchrony with an auditory signal. Proceedings of the Royal Society B, 273, 865-874.
Gallace, A., & Spence, C. (2006). Multisensory synesthetic interactions in the speeded classification of visual size. Perception & Psychophysics, 68 (7), 1191-1203.
Guski, R., & Troje N. F. (2003). Audiovisual phenomenal causality. Perception & Psychophysics, 65, 789-800.
Heron, J., Whitaker, D., McGraw, P. V., & Horoshenkov, K. V. (2007). Adaptation minimizes distance-related audiovisual delays. Journal of Vision, 7 (13):5, 1-8.
Howard, I. P., & Templeton, W. B. (1966). Human Spatial Orientation. Oxford, England: John Wiley & Sons.
Kanaya, S., & Yokosawa, K. (2011). Perceptual congruency of audio-visual speech affects ventriloquism with bilateral visual stimuli. Psychonomic Bulletin and Review, 18(1), 123-128.
Koelewijn, T. Bronkhorst, A. & Theeuwes, J. (2010). Attention and the multiple stages of multisensory integration: A review of audio-visual studies. Acta Psychologica, 134, 372-384.
Kohlrausch, A., & van de Par, S. (2000). Experimente zur Wahrnehmbarkeit von Asynchronie in audio-visuellen Stimuli [Experiments on the perception of asynchrony with audio-visual stimuli]. In Fortschritte der Akustik (DAGA 2000, pp. 316-317). Oldenburg: DEGA Geschaftstelle.
Kopinska, A. & Harris, L. R. (2004). Simultaneity constancy. Perception, 33, 1049-1060.
Lewald, J., & Guski, R. (2003). Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Cognitive Brain Research, 16 (3), 468-478.
Lewald, J., & Guski, R. (2004). Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception. Neuroscience Letters, 357, 119-122.
Marks, L. E. (1987). On cross-modal similarity: Auditory-visual interactions in speeded discrimination. Journal of Experimental Psychology: Human Perception and Performance, 13, 384-394.
McGurk, H., & MacDonald J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
Melara, R. D., & O’Brien, T. P. (1987). Interaction between synesthetically corresponding dimensions. Journal of Experimental Psychology: General, 116 (4), 323-336.
Michotte, A. (1946). La perception de la causalité. Louvain: Publications Universitaires. [English translation: The perception of causality. London: Methuen, 1963.]
Molholm, S., Ritter, W., Javitt, D. C. & Foxe, J. J. (2004). Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex, 14, 452-465.
Neuhoff, J. G. (2001). An adaptive bias in the perception of looming auditory motion. Ecological Psychology, 13 (2), 87-110.
Parise, C. V. & Spence, C. (2009). ‘When birds of a feather flock together’: Synesthetic correspondences modulate audio-visual integration in non-synesthetes. PLoS One, 4, e5664.
Patching, G. R., & Quinlan, P. T. (2002). Garner and congruence effects in the speeded classification of bimodal signals. Journal of Experimental Psychology: Human Perception and Performance, 28 (4), 755-775.
Radeau, M. (1985). Signal intensity, task context, and auditory-visual interactions. Perception, 14(5), 571-577.
Recanzone, G. H. (2003). Auditory influences on visual temporal rate perception. Journal of Neurophysiology, 89, 1078-1093.
Roach, N. W., Heron, J., & McGraw, P. V. (2006). Resolvin multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration. Proceedings of the Royal Society B, 273, 2159-2168.
Roseboom, W., Nishida, S. & Arnold, D. H. (2009). The sliding window of audio-visual simultaneity. Journal of Vision, 9, 1-8.
Scholl, B. J., & Nakayama, K. (2000, November). Contextual effects of the perception of causality. Poster presented at the annual meeting of the Psychonomic Society, New Orleans. [Abstract published in Abstracts of the Psychonomic Society, 5, 91].
Slutsky, D. A., & Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect. NeuroReport, 12 (1), 7-10.
Smith, L. B., & Sera, M. D. (1992). A developmental analysis of the polar structure of dimensions. Cognitive Psychology, 24 (1), 99-142.
Soetens, E. Maetens, K., Zeischka, P. & Henderickx, D. (2010). Congurnecy reversals in an accessory signal Simon task with auditory and visual stimuli. Acta Psychologia, 134, 391-397.
Soto-Faracao, S. and Alsisus, A. (2009). Deconstructing the McGurk-MacDonald illusion. Journal of Experimental Psychology: Human Perception and Performance, 35, 580-587.
Spence, C. (2011). Crossmodal correspondences: a tutorial review. Attention, Perception, and Psychophysics, 73 (4), 971-995.
Spence, C. & Squire, S. (2003). Multisensory integration: maintaining the perception of synchrony. Current Biology, 13, R519-R521.
Stein, B. E., & Stanford T. R. (2008). Multisensory integration: current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9, 255-266.
Stone, J. V., Hunkin, N. M., Porrill, J., Wood, R., Keeler, V., Beanland, M., Port, M., & Porter, N. R. (2001). When is now? Perception of simultaneity. Proceedings of the Royal Society B, 268, 31-38.
Sugita, Y., & Suzuki, Y. (2003). Audiovisual perception: implicit estimation of sound-arrival time. Nature, 421, 911.
Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.
Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Science, 14 (9), 400-410.
Teder-Sälejärvi, W. A., McDonald, J. J., Di Russo, F., Hillyard, S. A. (2002). An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Cognitive Brain Research, 14, 106-114.
van Wassenhove, V., Grant, K. W. & Poeppel, D. (2007). Temporal window of integration in auditory-visual speech perception. Neurophysiologica, 45, 598-607.
Vroomen, J., Keetels, M., de Gelder, B., & Bertelson, P. (2004). Recalibration of temporal order perception by exposure to audiovisual asynchrony. Cognitive Brain Research, 22 (1), 32-35.
Vroomen, J., & Keetels, M. (2010). Perception of intersensory synchrony: a tutorial review. Attention, Perception, & Psychophysics, 72 (4), 871-884.
Walker, P., Bremner, J. G., Mason, U., Spring, J., Matlock, K., Slater, A. & Johnson, S. P. (2010). Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences. Psychological Science, 21, 21-25.
Walsh, V. (2003). A theory of magnitude: common cortical metrics of time, space, and quantity. Trends in Cognitive Sciences, 7 (11), 483-488.
Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88, 638-667.
Zhu, X. R., Zhang, H. J., Wu, T. T., Luo, W. B. & Luo, Y. J. (2010). Emotional conflict occurs at an early stage: Evidence from the emotional face-word Stroop task. Neuroscience Letters, 478, 1-4.
Zmigrod, S., & Hommel, B. (2011). The relationship between feature binding and consciousness: evidence from asynchronous multi-modal stimuli. Consciousness and Cognition, available online first: doi: 10.1016/j.concog.2011.01.011.
Alais, D., & Carlile, S. (2005). Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound. Proceedings of the National Academy of Sciences of the United States of America, 102 (6), 2244-2247.
Arnold, D. H., Johnston, A., & Nishida, S. (2005). Timing sight and sound. Vision Research, 45, 1275-1284.
Bermant, R. I., & Welch, R. B. (1976). Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Perceptual and Motor Skills, 43, 487-493.
Bien, N., ten Oever, S., Goebel, R., & Sack, A. T. (2012). The sound of size: crossmodal binding in pitch-size synesthesia: a combined TMS, EEG and psychophysics study. NeuroImage, 59 (1), 663-672.
Burr, D., Banks, M. S., & Morrone, M. C. (2009). Auditory dominance over vision in the perception of interval duration. Experimental Brain Research,198 (1), 49-57.
Calvert, G. A., Spence, C., & Stein, B. E. (2004). The handbook of multisensory processing. Cambridge, MA: MIT Press.
Cappe, C., Thut, G., Romei, V., & Murray, M. M. (2009). Selective integration of auditory-visual looming cues by humans. Neuropsychologia, 47 (4), 1045-1052.
Cohen, J., MacWhinney, B., Flatt, M. & Provost, J. (1993). PsyScope: An interactive graphic system for designing and controlling experiments in the psychology laboratory using Macintosh computers. Behavior Research Methods, Instruments, and Computers, 25, 257-271.
Dixon, N. F., & Spitz, L. (1980). The detection of auditory visual desynchrony. Perception, 9, 719-721.
Engel, G. R., & Dougherty, W. G. (1971). Visual-auditory distance constancy. Nature, 234, 308.
Evans, K. K., & Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features. Journal of Vision, 10 (1), 1-12.
Fujisaki, W., Shimojo, S., Kashino, S. & Nishida, S. (2004). Recalibration of audio-visual simultaneity. Nature Neuroscience, 7, 773-778.
Fujisaki, W., Koene, A., Arnold, D., Johnston, A., & Nishida, S. (2006). Visual search for a target changing in synchrony with an auditory signal. Proceedings of the Royal Society B, 273, 865-874.
Gallace, A., & Spence, C. (2006). Multisensory synesthetic interactions in the speeded classification of visual size. Perception & Psychophysics, 68 (7), 1191-1203.
Guski, R., & Troje N. F. (2003). Audiovisual phenomenal causality. Perception & Psychophysics, 65, 789-800.
Heron, J., Whitaker, D., McGraw, P. V., & Horoshenkov, K. V. (2007). Adaptation minimizes distance-related audiovisual delays. Journal of Vision, 7 (13):5, 1-8.
Howard, I. P., & Templeton, W. B. (1966). Human Spatial Orientation. Oxford, England: John Wiley & Sons.
Kanaya, S., & Yokosawa, K. (2011). Perceptual congruency of audio-visual speech affects ventriloquism with bilateral visual stimuli. Psychonomic Bulletin and Review, 18(1), 123-128.
Koelewijn, T. Bronkhorst, A. & Theeuwes, J. (2010). Attention and the multiple stages of multisensory integration: A review of audio-visual studies. Acta Psychologica, 134, 372-384.
Kohlrausch, A., & van de Par, S. (2000). Experimente zur Wahrnehmbarkeit von Asynchronie in audio-visuellen Stimuli [Experiments on the perception of asynchrony with audio-visual stimuli]. In Fortschritte der Akustik (DAGA 2000, pp. 316-317). Oldenburg: DEGA Geschaftstelle.
Kopinska, A. & Harris, L. R. (2004). Simultaneity constancy. Perception, 33, 1049-1060.
Lewald, J., & Guski, R. (2003). Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli. Cognitive Brain Research, 16 (3), 468-478.
Lewald, J., & Guski, R. (2004). Auditory-visual temporal integration as a function of distance: no compensation for sound-transmission time in human perception. Neuroscience Letters, 357, 119-122.
Marks, L. E. (1987). On cross-modal similarity: Auditory-visual interactions in speeded discrimination. Journal of Experimental Psychology: Human Perception and Performance, 13, 384-394.
McGurk, H., & MacDonald J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.
Melara, R. D., & O’Brien, T. P. (1987). Interaction between synesthetically corresponding dimensions. Journal of Experimental Psychology: General, 116 (4), 323-336.
Michotte, A. (1946). La perception de la causalité. Louvain: Publications Universitaires. [English translation: The perception of causality. London: Methuen, 1963.]
Molholm, S., Ritter, W., Javitt, D. C. & Foxe, J. J. (2004). Multisensory visual-auditory object recognition in humans: A high-density electrical mapping study. Cerebral Cortex, 14, 452-465.
Neuhoff, J. G. (2001). An adaptive bias in the perception of looming auditory motion. Ecological Psychology, 13 (2), 87-110.
Parise, C. V. & Spence, C. (2009). ‘When birds of a feather flock together’: Synesthetic correspondences modulate audio-visual integration in non-synesthetes. PLoS One, 4, e5664.
Patching, G. R., & Quinlan, P. T. (2002). Garner and congruence effects in the speeded classification of bimodal signals. Journal of Experimental Psychology: Human Perception and Performance, 28 (4), 755-775.
Radeau, M. (1985). Signal intensity, task context, and auditory-visual interactions. Perception, 14(5), 571-577.
Recanzone, G. H. (2003). Auditory influences on visual temporal rate perception. Journal of Neurophysiology, 89, 1078-1093.
Roach, N. W., Heron, J., & McGraw, P. V. (2006). Resolvin multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration. Proceedings of the Royal Society B, 273, 2159-2168.
Roseboom, W., Nishida, S. & Arnold, D. H. (2009). The sliding window of audio-visual simultaneity. Journal of Vision, 9, 1-8.
Scholl, B. J., & Nakayama, K. (2000, November). Contextual effects of the perception of causality. Poster presented at the annual meeting of the Psychonomic Society, New Orleans. [Abstract published in Abstracts of the Psychonomic Society, 5, 91].
Slutsky, D. A., & Recanzone, G. H. (2001). Temporal and spatial dependency of the ventriloquism effect. NeuroReport, 12 (1), 7-10.
Smith, L. B., & Sera, M. D. (1992). A developmental analysis of the polar structure of dimensions. Cognitive Psychology, 24 (1), 99-142.
Soetens, E. Maetens, K., Zeischka, P. & Henderickx, D. (2010). Congurnecy reversals in an accessory signal Simon task with auditory and visual stimuli. Acta Psychologia, 134, 391-397.
Soto-Faracao, S. and Alsisus, A. (2009). Deconstructing the McGurk-MacDonald illusion. Journal of Experimental Psychology: Human Perception and Performance, 35, 580-587.
Spence, C. (2011). Crossmodal correspondences: a tutorial review. Attention, Perception, and Psychophysics, 73 (4), 971-995.
Spence, C. & Squire, S. (2003). Multisensory integration: maintaining the perception of synchrony. Current Biology, 13, R519-R521.
Stein, B. E., & Stanford T. R. (2008). Multisensory integration: current issues from the perspective of the single neuron. Nature Reviews Neuroscience, 9, 255-266.
Stone, J. V., Hunkin, N. M., Porrill, J., Wood, R., Keeler, V., Beanland, M., Port, M., & Porter, N. R. (2001). When is now? Perception of simultaneity. Proceedings of the Royal Society B, 268, 31-38.
Sugita, Y., & Suzuki, Y. (2003). Audiovisual perception: implicit estimation of sound-arrival time. Nature, 421, 911.
Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.
Talsma, D., Senkowski, D., Soto-Faraco, S., & Woldorff, M. G. (2010). The multifaceted interplay between attention and multisensory integration. Trends in Cognitive Science, 14 (9), 400-410.
Teder-Sälejärvi, W. A., McDonald, J. J., Di Russo, F., Hillyard, S. A. (2002). An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Cognitive Brain Research, 14, 106-114.
van Wassenhove, V., Grant, K. W. & Poeppel, D. (2007). Temporal window of integration in auditory-visual speech perception. Neurophysiologica, 45, 598-607.
Vroomen, J., Keetels, M., de Gelder, B., & Bertelson, P. (2004). Recalibration of temporal order perception by exposure to audiovisual asynchrony. Cognitive Brain Research, 22 (1), 32-35.
Vroomen, J., & Keetels, M. (2010). Perception of intersensory synchrony: a tutorial review. Attention, Perception, & Psychophysics, 72 (4), 871-884.
Walker, P., Bremner, J. G., Mason, U., Spring, J., Matlock, K., Slater, A. & Johnson, S. P. (2010). Preverbal infants’ sensitivity to synaesthetic cross-modality correspondences. Psychological Science, 21, 21-25.
Walsh, V. (2003). A theory of magnitude: common cortical metrics of time, space, and quantity. Trends in Cognitive Sciences, 7 (11), 483-488.
Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88, 638-667.
Zhu, X. R., Zhang, H. J., Wu, T. T., Luo, W. B. & Luo, Y. J. (2010). Emotional conflict occurs at an early stage: Evidence from the emotional face-word Stroop task. Neuroscience Letters, 478, 1-4.
Zmigrod, S., & Hommel, B. (2011). The relationship between feature binding and consciousness: evidence from asynchronous multi-modal stimuli. Consciousness and Cognition, available online first: doi: 10.1016/j.concog.2011.01.011.