When exposed to acoustic and visual stimuli, human observers may perceive them as originating either from a common or from distinct sources. Audiovisual perception includes both integration and segregation effects whose importance in the final percept depends on the stimuli spatio-temporal synchrony. This phenomenon has been studied through perception tasks, where a stimulus in one modality may be perturbed by another stimulus in the other modality [1-2]. The authors have inferred some probabilistic relationships between the emitted and the reported stimulus location.

We have performed a similar experiment to study and model these audiovisual integration and segregation phenomena. Subjects, seated in a dark room, were exposed to acoustic and visual stimuli presented either alone (unimodal case) or together (bimodal case). In the latter case, the stimuli co-occurred in time but not necessary in space. The subjects had to report the location of the visual or acoustic stimulus using a manual pointer connected to a potentiometer.

The singularity of our work was to use Bayesian networks [3] to build a consistent model where a graph structure is learned from the data. This structure states the dependencies between random variables modeling the events (such as the emitted and perceived stimulus locations). The corresponding probabilistic relationships are then inferred.

[1] Konrad P. Körding, Ulrich Beierholm, Wei Ji Ma, Steven Quartz, Joshua B. Tenebaum, and Ladan Shams. Causal inference in multisensory perception. PLoS ONE, 2(9):e943, September 2007. doi:10.1371/journal.pone.0000943.

[2] Neil W. Roach, James Heron, and McGraw Paul V. Resolving multisensory conflict: a strategy for balancing the costs and benefits of audio-visual integration. In Proceedings of the Royal Society B: Biological Sciences, volume 273, pages 2159–2168, June 2006. doi: 10.1098/rspb.2006.3578.

[3] Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.