I n t e r n a t i o n a l   M u l t i s e n s o r y    R e s e a r c h    F o r u m 2ND ANNUAL  MULTISENSORY RESEARCH CONFERENCE

 
Conference Abstracts
 
Abstract Numbers: 1-9   10-19   20-29   30-39   40-49   50-59
Keynote Abstracts: Massaro   Stein

conference program
imrf conference main page

 

19	Crossmodal stimulation through the looking glass: cognitive remapping of visual-tactile information in the normal and damaged brain.
	Angelo Maravita
	The distance of visual stimuli from the skin surface constrains visual-tactile interactions. Electrophysiological studies in monkeys show that  the response of cross-modal visual-tactile cells is stronger for a visual stimulus presented close to their tactile receptive field. This makes functional sense given that in condition of natural stimulation, related visual-tactile information is normally conveyed with spatial-temporal correspondence. However, there is a common situation where visual stimuli appear far away in space despite being close to the skin surface: That is when we observe ourselves in mirrors, while acting on our own body (combing, etc.). In these cases, visual stimuli near our body (e.g. the comb in our hair) project the image of distant objects, as if placed through the looking glass". How does the brain cope with such paradoxical sensory information?  We tested normal observers in a visual-tactile interference paradigm. In this paradigm usually tactile judgements are disrupted more by visual distractors close to the stimulated hand, than by far distractors. In one condition (Mirror) we presented visual distractors near the hands while subjects observed their own hands and the distractors reflected in a mirror. In another condition (Box) subjects observed, through the half silvered mirror, the contents of a box, which comprised visual distractors and two stuffed rubber hands, placed far away, at the exact position consistent with the reflections of visual distractors and subjectÃs hands in the Mirror condition. Visual interference was stronger in the Mirror condition than in the Box condition.  A similar finding was observed when testing visual-tactile extinction in a right brain damaged patient. The effect of a right, ipsilesional visual stimulus in extinguishing awareness of a left, contralesional touch was stronger when the former stimulus was near to the ipsilesional hand, but observed distantly via a mirror, than when it was directly observed in the distance (box condition). These results suggest that visual stimuli near the hands, although optically appearing far away when observed in a mirror (through the looking glass"), are computed by the brain as being in peripersonal space, near to the corresponding body part, thus allowing strong visual-tactile interactions.

 

20	Crossmodal bias in audio-visual perception of emotion
	Bea De Gelder

21	A novel form of multisensory neuron resulting from excitatory-inhibitory convergence
	Lisa R. Dehner, H.Ruth Clemo, and M. Alex Meredith
	Attention to stimuli from one modality can result in reduced responses to stimuli from other modalities.ð These effects cannot be attributed to the activity ofð neurons which are excited by more than one modality.ð Instead, convergence of excitatory inputs from one modality with inhibitory afferents from another would produce a multisensory neuron in which activity evoked by one modality could be modulated (through inhibition and disinhibition) by the other.ð This novel form of multisensory convergence was examined in the anterior ectosylvian sulcal cortices of the cat. Neuroanatomical and antidromic stimulation techniques revealed that auditory Field AES (FAES) reliably projects to somatosensory (SIV), but electrical stimulation of FAES failed to orthodromically excite any SIV neuron.ð However, when FAES activation was paired with effective tactile stimulation, the responses of a majority of SIV neurons were suppressed.ð This suppressive effect was not achieved by stimulation of the adjacent primary auditory cortices, nor were somatosensory responses in the nearby SII regions affected by FAES activation.ð Ultimately, the application of bicuculline methiodide, the antagonist to the inhibitory neurotransmitter GABA- A, blocked the suppressive effects of FAES stimulation on SIV neurons, indicating the presence of active inhibition within the FAES-SIV circuit.These data are consistent with a cross-modal projection between higher-level auditory and somatosensory areas.ð Rather than exhibiting the familiar excitatory bimodal properties, manyð SIV neurons show a here-to-for undescribed form of excitatory-inhibitory convergence that ultimately results in modulation of activity in one modality by inhibition (and disinhibition) from another.ð In this fashion, attention to auditory stimuli could reduce SIV responses to spatially/temporally coincident tactile cues. Supported byð NS39460 and by the Human Frontiers Science Program RGO174-98.

22	Auditory-visual convergence in the primate prefrontal cortex
	Liz Romanski
	Our ability to recognize and integrate auditory and visual stimuli is the basis for many cognitive processes but is especially essential in meaningful communication.  Although many brain regions contribute to recognition and integration of sensory signals, the frontal lobes both receive a multitude of afferents from sensory association areas and have influence over a wide region of the nervous system to govern behavior. Furthermore, the frontal lobes are special in that they have been associated with language processes, working memory, planning, and reasoning, which all depend on the recognition and integration of a vast network of signals. The dorsolateral prefrontal cortex, including Brodman's areas 8 and 46, has been previously associated with visuo-spatial processing (Goldman-Rakic, 1987).  However, recent studies indicate that it may also be involved in auditory spatial processing (Bushara et al., 1999).  The ventrolateral prefrontal cortex, referred to as the inferior convexity, is most often associated with language processing in the human brain.  Neuroimaging studies indicate that this area is also involved in visual working memory and therefore may be involved in multisensory integration. Recent analysis of the inferior convexity (IFC) of the prefrontal cortex in non-human primates has revealed a non-spatial visual processing region. In these studies, visual neurons with receptive fields that included the fovea were shown to have robust and highly specific responses to pictures of objects and faces (O'Scalaidhe et al., 1997).  Since the macaque inferior convexity is the recipient of auditory afferents (Romanski et al., 1999a, 199b) and the inferior frontal lobe in the human brain is involved in language processing we searched for an auditory processing region in the macaque IFC. We found that there is, in fact, an auditory receptive area with neurons that are responsive to complex sounds including species-specific vocalizations.  The fact that there are juxtaposed and overlapping areas of visual and auditory processing in the IFC indicates a possible role in multimodal sensory integration. Furthermore, the overlap of a visual face-processing area and an auditory area where neurons are activated by vocalizations suggests that this region may integrate multisensory stimuli for the purpose of communication and recognition. Experiments aimed at testing whether single neurons in the IFC are multimodal are currently underway.  Preliminary anatomical and physiological evidence suggests that the IFC of the primate frontal lobe has separate and overlapping domains for the processing of auditory, visual and multimodal information. Further analysis of the IFC of the frontal lobe in non-human primates will provide us with detailed knowledge of the neuronal networks and ensembles that support complex cognitive processes including mulitimodal communication.

23	Visual Imagery in tactile perception
	Krish Sathian
	Psychophysical studies implicate visual imagery in certain aspects of tactile perception. Functional imaging and event-related potential studies demonstrate that visual cortical areas are active during tactile judgments of certain object properties. Moreover, these visual areas are necessary for optimal tactile performance, as revealed by transient disruption of cortical activity using transcranial magnetic stimulation. These findings in normally sighted subjects indicate that there are intimate cross-modal links between touch and vision.

24	Crossmodal links in spatial attention
	Charles Spence
	Most research in cognitive psychology on selective attention has implicitly assumed that each sensory modality can be considered in isolation, and has thus overlooked many issues concerning crossmodal links in attention. In daily life, however, our spatial attention often has to be co-ordinated across several modalities simultaneously. Recent research has revealed the existence of extensive crossmodal links in covert spatial attention between audition, vision, touch, and even olfaction. I will report a number of studies in normal people that highlight some of the constraints inherent in our ability to selectively focus and divide our attention across different modalities and locations simultaneously. Further studies demonstrate that these crossmodal links in spatial attention are maintained under conditions of receptor misalignment, such as when the eyes are deviated with respect to the head, or the hands are crossed over. Finally, I will show how multisensory integration can break down in the split-brainÃ, and in normal people receiving repetitive transcranial magnetic stimulation (rTMS).

25	Computational approaches to multisensory integration: an overview
	Christian Scheier
	There has been a significant increase of empirical work on multisensory integration over the past years. Psychophysical and neurophysiological experiments are beginning to reveal some of the principles the brain uses to integrate information across sensory modalities. We can now start to ask which computational mechanisms best describe these principles. This talk will present an overview on the computational approaches to multisensory integration that have recently been proposed. We will focus on the principles that have been suggested for (a) coordinate transformations between sensory channels, (b) development and alignment of multisensory maps, and (c) spatial and temporal integration of multisensory information. We will cover a rather wide range of application areas ranging from integration in the superior colliculus to speech perception and sensor fusion in robots. Approaches discussed include race models, neural network models, Baysian models, and dynamical systems models. We will conclude by pointing out some of the issues that need to be addressed in future computational work on multisensory integration

26	Multiple forms of sensory integration: modeling the fusion process
	John Jeka, Kelvon Oie and Tim Kiemel
	Different tasks requiring multiple forms of sensory information may invoke fundamentally different processes to solve the fusion problem. For example, reaching to a target specified by more than one sensory source requires estimating position of the target based upon sensory information that is often discontinuous (e.g., flash of light, brief tone). The problem is often posed as where sensory fusion occurs along the processing stream to form a reliable estimate, which is known to be influenced by factors such as relative stimulus intensity, probability, S-R compatibility, etc.ð By contrast, fusion of multisensory information for human postural control requires online updating of estimates of a moving center of mass; a stochastic process based upon sensory information that is continuously available. An important issue for postural control is the nature of the fusion process. Is it inherently linear/nonlinear or does the process of fusion depend upon sensory context? We present results from two experiments which, when compared to the predictions of a 3rd-order model for postural control and its coupling to sensory information, suggest multiple forms of fusion. Subjects stood in front of large-screen visual display while lightly touching a rigid surface with the right index fingertip. In Experiment 1, anti-phase, oscillatory, visual and somatosensory stimuli were presented at a constant frequency (0.2 Hz). The amplitudes of the stimuli were inversely scaled relative to each other in 0.2 cm increments over a range of 0.0-2.0 cm. Across subjects, three forms of fusion were observed; linear averaging, reweighting and switching, indicating that the fusion process is not entirely dependent on sensory context.ð In Experiment 2, we investigated the fusion process when visual and touch stimuli were oscillated at different frequencies and different relative amplitudes. Preliminary analysis of fitted model parameters showed that changes in the gain response across condition were due primarily to changes in the weighting of sensory inputs and not due to changes in control system parameters. Results from these experiments suggest that the nervous system fuses multiple sensory inputs through both linear and nonlinear processes.

27	What are the fundamental, conceptually challenging problems for understanding multi-sensory fusion?
	Gregor Schoner
	This talk will concentrate on three main points.ð (1) Each individual source of sensory information may carry processes of interpolation or model-based estimation of parameters that characterize the physical source of stimulation. ð(2) Different sources of sensory information may differ with respect to these processes, including, in particular, the reference frames relative to which such interpolation/estimation takes place.ð (3) The fusion process may combine different estimates, but may also involve decision making in which selected sources of sensory specification are neglected or suppressed.ð These issues are discussed from the perspective of a potentially unifying dynamic field theory.ð Other frameworks originating in the engineering discipline of data and decision fusion are also referred to.

28	Modelling Multisensory Enhancement in the Superior Colliculus
	Tom Anastasio
	The deep layers of the superior colliculus (DSC) integrate multisensory input and trigger orienting responses toward salient targets. DSC neurons are organized topographically, and the maps representing different sensory modalities are in register. Some individual DSC neurons receive multimodal sensory input, while others are unimodal.Multisensory enhancement (MSE) is a form of multisensory integration in which the neural response to a stimulus of one modality is augmented by a stimulus of another modality. Smaller unimodal responses are associated with larger percentages of multisensory enhancement -- a property called inverse effectiveness. We have developed a probabilistic model of MSE, based on the hypothesis that a deep SC neuron uses its sensory inputs (multimodal or unimodal) to compute the conditional probability that a target is present in its receptive field. We model the sensory inputs to the deep SC as random variables, and cast the computation function in terms of Bayes' rule. The Bayes' rule model reproduces MSE, and suggests that inverse effectiveness results because the increase in target probability, due to the integration of multisensory inputs, is larger when the unimodal responses are smaller. The difference between the input probability distributions when the target is present and when it is absent can be quantified using the Kullback/Leibler difference (D). Consistent with intuition, the amount of target information provided by the inputs increases as D increases, and information gain increases faster for multimodal than for unimodal inputs. However, as D increases further, information gain plateaus at the same level in both the multimodal and unimodal cases. Unimodal deep SC neurons may be those that receive high D input of one modality and have no need for input of another modality. The Bayes' rule model specifies that the amount of MSE exhibited by a deep SC neuron should depend upon the prior probability (unconditioned on sensory input) that a target will appear in its receptive field. This prior is primarily dependent upon the location of the deep SC neuron's receptive field. The model predicts that deep SC neurons with more peripheral receptive fields should exhibit larger amounts of MSE.

29	Information supporting direct multimodal perception
	Thomas Stoffregen and Benoit G. Bardy
	In traditional or Helmholtzian theories of perception, the patterns of ambient energy (e. g., light, sound), that are available to perceptual systems are assumed to bear an ambiguous relation to reality. Given that perception generally is accurate, the assumed ambiguity in the stimulus implies that perceptual accuracy depends upon inferential or associative processing in the head. An alternative view is offered by the Ecological Approach to Perception and Action, in which it is argued that patterns of ambient energy bear an unambiguous relation to reality, such that these patterns specify reality. If this is true then, in principle, perception can be accurate without the intervention of inferential or associative processes ; psychological activity may concentrate on interactions with the (accurately perceived) world, rather than on the relation between sensory stimulation and reality. Students of the Ecological Approach have identified (both mathematically and through behavioral research) many patterns of ambient energy that, it is claimed, specify various aspects of reality. This effort has been sufficiently successful that it is now widely accepted that certain patterns are unambiguously related to particular physical events ; for example, that optical flow specifies self-motion. This effort has been concentrated on patterns that exist within individual forms of ambient energy, such as the optic array, the acoustic array, and so on. Such research accepts the traditional assumption that perception takes place in separate modalities (e. g., vision, audition). In the present contribution we present a novel analysis of the information available to perceptual systems, that is, of the stimulus. We point out the existence of patterns that extend across different forms of ambient energy, and we argue that these patterns provide information that is essential for the perception and control of ordinary behavior. These patterns make it possible, in principle, for " multimodal " perception to be accurate without inferential processing. This possibility would require a radical change in our understanding of perceptual systems, including rejection of the assumption that vision, audition, and so on exist as separate sensory channels or perceptual systems.

 

30	Prior Entry: Previous confounds and novel findings using an orthogonal design
	David I. Shore, Charles Spence and R.M. Klein
	Despite nearly two centuries of research, the question of whether attending to a sensory modality speeds the perception of stimuli in that modality has yet to be resolved. We highlight several confounds inherent in this previous research, and then report a series of four experiments designed to investigate the effects of attending to a particular sensory modality or spatial location on temporal order judgments (TOJs) using a novel methodology. Participants were presented with pairs of visual and tactile stimuli from either the left and/or right of fixation at varying stimulus onset asynchronies (SOAs), and were required to make unspeeded TOJs regarding which stimulus appeared first. Typically, visual stimuli had to be presented before tactile stimuli for them to be judged as simultaneous. Additionally, accuracy was significantly better when the stimuli were presented from different positions rather than from the same position. To be perceived as simultaneous, visual stimuli had to lead by a greater temporal interval when tactile stimuli were attended (Experiment 2A), and by less when visual stimuli were attended (Experiment 2B). When target stimuli were more likely on the left than right, participants judged the left stimulus as having been presented earlier than the right compared to when attention was spatially-divided or when participants attended to the right side (Experiment 3 and 4). These results provide the first conclusive evidence for the existence of crossmodal prior entry; they also provide support for previous claims of an attentional bias toward the visual modality, and also for a bias toward the right side of space.

31	Multisensory integration subserving orienting behavior
	Douglas Munoz, Brian Corneil, Andrew Bell, Alex Meredith, John van Opstal
	Gaze shifts are coordinated movements of the eyes and head that rapidly reorient the visual axis to a target of interest. To gain insight into the processes of multisensory integration, we studied the influences of competing visual and auditory stimuli on horizontal gaze shifts in humans in a variety of behavioral conditions. Gaze shifts were made to visual or auditory targets in the presence of either an irrelevant visual or auditory cue. Within an experiment, the target and irrelevant cue were either aligned (enhancer condition) or misaligned (distractor condition) in space. We manipulated the temporal asynchrony between onset of visual and auditory stimuli, the spatial location of stimuli, and the state of visual fixation at the time of stimulus onset. We then compared subject performance in enhancer and distractor conditions and measured reaction times and the frequency of incorrect gaze shifts made to the distractors. Our results reveal that, in addition to the spatial and temporal register of the stimuli, the experimental context in which the stimuli are presented influences multisensory integration. First, subjects had more difficulty suppressing incorrect movements to the distractors when the fixation target was extinguished prior to target onset. Second, an irrelevant auditory cue influenced gaze shifts to visual targets differently than an irrelevant visual cue influenced gaze shifts to auditory targets. The above studies have now been expanded upon by examining neural activity in the superior colliculus of monkeys trained to orient to either visual, auditory, or combined stimuli. The activity of saccade-related neurons was modulated by the nature of the saccade target and state of visual fixation. Further, the relative positions and temporal asynchrony of the visual and auditory stimuli also had a strong influence on the activity of these neurons. In terms of behavior, combination of the two stimuli in the enhancer condition resulted in a decrease in saccadic reaction time, similar to what we observed in humans.Our results revealed that the processes of fixation disengagement, multisensory integration, and target selection are interrelated and arise, at least in part, from interactions within the superior colliculus.

 

32	K e y n o t e
	Multisensory Research: A summary, integration and future directions
	Barry Stein
	This talk will provide an overview of issues in Multisensory Science and the proceedings of this year's annual meeting, in addition to perspectives on early questions about Multisensory Integration and the current state of the field.

33	Is binding required to conjoin visual and tactile features?
	Christopher T. Lovelace and Peter Grossenbacher
	Discrete visual features like color and shape are processed in largely separate, parallel sensory channels, but are integrated ("bound") in perception. Behavioral studies suggest that feature binding occurs only after parallel processing of each feature is complete. Responses to visual targets identified by conjunctions of multiple features (e.g., line orientation and color) are slower than responses to targets identified by a single feature (the implication being that the added time is necessitated by the binding process). The current study sought to determine whether responses to targets identified by a conjunction of features in separate sense modalities are slower than responses to features in each sense. Twelve subjects discriminated low and high frequencies using vision and touch. A brief (151 ms) multisensory stimulus was presented on each trial, and consisted of a fingertip vibration (200 or 425 Hz) and a 5 deg. wide sinusoidal luminance grating (2 or 8 cpd). Frequency in vision and touch were varied independently across trials (ISI = 800 - 1200 ms). Subjects made speeded responses only to targets, which were defined by frequency in either a single modality (Feature) or both modalities (Conjunction). Each frequency and combination of frequencies was a target in separate trial blocks (target probability = 0.25). If identification of Conjunction targets requires serial binding of the visual and tactual features, then reaction times (RTs) to Conjunction targets should be longer than RTs to Feature targets. Virtually all individual median RTs to Conjunction targets were at least 25 ms larger than to Feature targets, suggesting that identification of multisensory feature conjunctions is a serial process. In addition, there was a robust frequency correspondence effect between the vibrotactile and visuospatial features. Responses to vibrations were faster and more accurate when the vibration frequency matched that of the simultaneous (but irrelevant) luminance grating than when the frequencies did not match. Responses to luminance gratings were unaffected by vibration frequency. These results suggest that a) detection of conjunctions of visual and tactual features may depend on a serial binding process, and b) automatic processing of visual frequency influences tactual frequency discrimination.

34	Visual - tactile spatial interaction in a focused-attention task
	Adele Diederich and Hans Colonius
	Visual (LED) and tactile stimuli (shaker, vibration to palms) were simultaneously presented at various spatial locations. In a focused attention paradigm with visual targets and tactile distractors, saccadic response times (SRT) were shown to depend on spatial distance between the stimuli. Three experiments are reported. 1) Visual stimuli were presented 10 to the left or right of the central fixation point; tactile stimuli were presented 10, 50, 70, or 110 from central fixation (ipsi- or contra-lateral with respect to the LED). 2) Tactile stimuli were presented 10 to the left or right of central fixation; and visual stimuli were either presented 10, 50, or 70 from central fixation (ipsi- or contra-lateral with respect to the shaker). 3.) Visual and tactile stimuli were presented at the same locations 10, 50, or 70 from central fixation. Cross-modal interaction (SRT facilitation) could be observed as a function of the eccentricity of the stimuli, depending on both spatial distance and hand position.

35	Gating of afferent activity in an Aplysia mechanoafferent neuron
	Elizabeth Cropper, C. G. Evans, J. Jing, S. C. Rosen, I. Kupfermann
	The Aplysia mechanoafferent neuron B21 is peripherally activated during both phases of ingestive motor programs (i.e., during radula opening/protraction (O/P) and radula closing/retraction/closing (C/R)).ðO/P and C/R are mediated by different groups of cells and are antagonistic. This suggests that afferent transmission in B21 is likely to be regulated so its ability to excite specific follower neurons will be phase-dependent.ðFor example, B21 is not likely to strongly excite a C/R follower throughout both O/P and C/R.ðOur experiments seek to characterize mechanisms that regulate afferent transmission in B21.ðDuring C/R B21 receives excitatory synaptic input from other neurons.ðWhen it is depolarized, peripherally initiated spikes in B21 do produce EPSPs in a follower neuron (B8).ðIn contrast, when B21 is at its resting membrane potential afferent activity is not transmitted to B8.ðThus, central depolarization 'gates in' B21 input to B8.ðB21 is a bipolar neuron.ðOne branch innervates the periphery (i.e., acts as the input branch in the context) and the other branch makes contact with B8 (i.e., acts as the output branch).ðWhen B21 is at its resting potential spikes recorded from the output branch are smaller than those recorded from the input branch or the B21 soma. More specifically, spikes in the output branch are quite small (e.g., 20 mV) thus are unlikely to trigger transmitter release.ðIn contrast, when B21 is centrally depolarized spikes in the output branch become full size.ðThus, we have characterized a mechanism that gates in afferent activity in B21 during C/R.ðTo determine whether afferent activity is actively 'gated out' during O/P we recorded from B21 during ingestive motor programs and found that IPSPs were observed during O/P. In part these IPSPs are due to activity of the O/P interneuron B52.ðB52 makes a monosynaptic connection with B21, and in otherwise quiescent preparations B52 stimulation dramatically decreases the size of spikes in the output branch of B21.ðMoreover B52 stimulation decreases the size of B21-induced EPSPs in B8. In conclusion, we have characterized a mechanism that regulates the transmission of afferent activity in a functionally defined context.

36	The effect of sound intensity and noise level on audiovisual speech perception
	Kaisa Tiippana, Mikko Sams and Riikka Mñttñnen
	Audiovisual speech perception was studied at various sound intensities and noise levels.ðThe stimuli were meaningless words /aka/, /apa/, /ata/ uttered by a female Finnish talker, presented both audiovisually and unimodally.ðFor congruent audiovisual stimuli, the auditory and visual stimuli were the same.ðFor incongruent audiovisual stimuli, the auditory and visual stimuli were different.ðIncongruent stimuli typically show the so-called McGurk effect, i.e. alteration of the speech percept from that produced by the auditory stimulus alone [1].ðThe signal-to-noise ratio (SNR) was varied either by changing the sound intensity (24 to 72 dB in 6 dB steps), or by adding white noise to the signals (SNRs of -12, -6, 0 and +6 dB).ðThe aims of this study were: (i) To determine whether the two manipulations of the acoustic SNR have a similar effect on audiovisual speech perception.ðIt has been reported that the McGurk effect becomes stronger as the auditory signal becomes louder [2].ðOn the other hand, there is evidence that the number of McGurk responses is greater with more auditory noise [3].ðWe wanted to clarify this discrepancy.ð(ii) To study whether congruent and incongruent stimuli are similarly affected by degradations in acoustic speech quality. (iii) To provide extensive data for testing models of audiovisual speech perception.ðThe consonant recognition results of 22 subjects with normal hearing and vision showed the following trends:ð(i) Both SNR manipulations had a similar effect on the subjects' responses.ðFor example, the McGurk effect became stronger as sound intensity decreased and as noise level increased, i.e. as the SNR decreased.ð(ii) For congruent stimuli, adding visual speech had a greater effect on the number of correct responses at lower SNRs, when compared to unimodal conditions.ðFor incongruent stimuli, the number of visual responses increased as the SNR decreased.ðThese results can be interpreted to indicate that the visual influence becomes greater for both congruent and incongruent stimuli as the SNR decreases.ð(iii) The first step in model testing will be fitting the Fuzzy Logical Model of Perception [4] to our results. References: [1]ð McGurk & MacDonald (1976). Hearing lips and seeing voices. Nature, 264, 746-748. [2]ð Kuhl, Green & Meltzoff (1988). Factors affecting the integration of auditory and visual information in speech: the level effect. Meeting of the Acoustical Society of America, Seattle. [3]ð Fixmer & Hawkins (1998). The influence of quality of information on the McGurk effect. Auditory-Visual Speech Processing Conference, Terrigal, Australia. [4]ð Massaro (1998). Perceiving talking faces. MIT Press, Cambridge, Massachusetts.

37	Is there a difference D between the amount of time required for auditory and visual stimuli to reach conscious awareness?
	James V. Stone, N.M. Hunkin, J. Porrill, R. Wood, V. Keeler, M. Beanland, M. Port and N.R. Porter
	This paper addresses the following question: is there a difference D between the amount of time required for auditory and visual stimuli to reach conscious awareness?ðOn each of 1000 trials, each of 23 observers was presented with a light and a sound, separated by a variable stimulus onset asynchrony (SOA) between -250ms (sound first) and +250ms (light first).ðThe light was a red LED (11cd/square metre), and the sound was a 1kHz square wave (71dB) presented through headphones. An observer indicated if the sound and light came on simultaneously by pressing one of two (yes/no) response keys, at which point both stimuli terminated.ðThe SOA most likely to yield affirmative responses was defined as the point of subjective simultaneity (PSS). The PSS for each observer was estimated as follows. The probability of an affirmative response described an inverted U-shaped curve as the SOA varied between -250ms and +250ms.ðThis inverted U-shaped curve was fitted with a Gaussian function using maximum likelihood estimation (MLE), and the PSS was taken to be the mean (peak) of the fitted Gaussian function.ðThe resultant observer-specific PSS values varied between -21ms (sem=4.5ms) (i.e. sound 21ms before light) and +150ms (sem=17.4ms) (i.e. light 150ms before sound) for different observers.ðAs acontrol, the difference RTd in simple reaction times (RT) to single audio and visual stimuli was also measured.ðThe PSS and RTd values have different means (t=2.10, p=0.052), and are uncorrelated.ðThe stability ofPSS and RTd values was evaluated by testing observers twice, with test sessions at least 24 hours apart.ðThe resultant pairs of PSS values, but not corresponding RTd values, are highly correlated (r=0.954, p=0.003). Together these results suggests that PSS values, but not RTd vales, are stable and observer-specific.ðThe implications of these findings for the perception of multisensory events are discussed.

38	Similar densities of GABA-ergic neurons in visual, auditory and somatosensory 'association' cortices of the cat
	H. Ruth Clemo and M. Alex Meredith
	It is well known that cortical receptive field properties are shaped byinhibitory processes, regardless of the sensory modality involved. Despite the physiological and perceptual distinctions among the different sensory modalities, the density of identified inhibitory (GABA-ergic) neurons is very similar across the different primary sensory cortices. However, receptive field properties become progressively more complex at higher cortical levels, and it is not known whether the inhibitory contributions to these higher-order processes requires larger , or different, proportions of GABA- ergic neurons.ð This possibility was examined in the cortices surrounding the AnteriorEctosylvian Sulcus (AES) which contains the higher order visual (AEV),somatosensory (SIV) and auditory (Field AES) representations. Coronal sections through the AES cortices were derived from 5 adultcats.ð Alternate sections were reacted for GABA, GAD65 and GAD67 using standard immunocytochemical procedures with silver-gold intensification. Tissue-outlines and immuno-positive neuron locations were plotted using a Nikon Optiphot with a computer-controlled stage; the number of labeled neurons was tabulated and the tissue area calculated. Measurements were taken from across the full thickness of the cortex (all laminae, I-VI)within AEV, SIV,ð and Field AES as well as from the adjacent gyralregions of AI/AII and SII.ð The mean density (neurons/mm2) and standard deviation of GABA-positive neurons was calculated for each cortical field: AEV =ð 92.1 ± 35.1;ð SIV =ð 103 ± 39.3;ð Field AES = 99.2ð ±54.3; AI/AII =ð 88.5 ± 24.9; and SII =ð 89.4 ± 35.1.ðð None of these values differed significantly from the other, nor did they differ dramatically from those obtained from the different primary sensory cortices. These data demonstrate that the concentration of inhibitory, GABA-ergicneurons within a given sensory cortex does not vary according to hierarchical level or sensory modality. This suggests that hierarchicalor modality-specific distinctions among receptive field properties result from other factors, perhaps such as differences in morphology of GABA-ergicneurons,ð their laminar distribution, or the density and distribution of GABA-ergic terminals.Supported byð NS39460 and by the Human Frontiers Science ProgramRGO174-98.

39	Tactile 'Inhibition-of-Return' (IOR) in congenitally blind humans
	Brigitte Roeder, Charles Spence, and Frank Roesler
	Our attentional systems orient reflexively to novel environmental stimuli. A period of inhibition, known as 'inhibition-of-return' (IOR), typically follows such attentive orienting. IOR has been demonstrated to affect the processing of visual, auditory and tactile stimuli, and also occurs crossmodally. IOR has been linked to the eye movement system and provides a tagging mechanism that prevents perseveration and hence facilitates attentional search. We examined whether the capacity for oculomotor control is a necessary prerequisite for the generation of IOR. We tested a group of 11 congenitally blind people who have either no control of their eye movements or no eyes at all, and 13 blindfolded sighted controls in a tactile IOR paradigm. Tactile targets requiring a speeded detection response were presented to the left or right index fingers, and were preceded by tactile cues occurring at either the same or opposite finger (ipsilateral vs. contralateral trials respectively). Both the sighted and congenitally blind groups demonstrated tactile IOR (i.e., longer target detection latencies on ipsilateral trials than on contralateral cue trials). Tactile IOR was also demonstrated in the blind participant without eyes. Overall mean target detection times were faster in the blind than in the sighted group. These results contradict the view that IOR depends on the control of eye movements, suggesting instead that IOR may reflect, at least in part, a more general inhibition of the orienting response. In addition, the shorter target detection times of the blind (compared to sighted) point to crossmodal compensation due to blindness.

 

40	Crossmodal Dynamic Capture
	Salvador Soto-Faraco, Alan Kingstone and Charles Spence
	We introduce a new crossmodal illusion in which the perceived direction of movement of a stimulus in one modality is strongly modulated by the direction of movement of a stimulus in a different modality. Participants were asked to determine the direction of movement (leftward vs. rightward) of two successive sounds (pure tones of 50 ms duration, 100 ms ISI). Participants performed at chance when a pair of simultaneous light flashes moved in the opposite direction to the sounds. However, performance was at ceiling when either: 1) No flashes were presented; 2) The lights moved in the same direction as the sounds; or 3) The visual and auditory stimuli were desynchronised, regardless of their respective direction of movement (i.e., same or different). These results show that crossmodal dynamic capture is not produced by cognitive or 'late' response bias factors. Furthermore, ventriloquism (capture) effects were significantly weaker in situations where no motion was perceived (i.e., static displays), demonstrating the crucial role of motion informtion in modulating our effects. These dynamic capture effects are consistent with the view that motion information across different sensory modalities is integrated automatically prior to perception.

41	EEG evidence for early audio-visual integration in the human brain
	Klaus Linkenkaer-Hansen, J. Matias Palva, Kai Kaila, and Risto J. Ilmoniemi
	A recent EEG-study in humans indicates that the processing streams of auditory and visual information interact already at 40 ms after stimulus onset [1]. Non-linear interaction between the processing streams of different stimuli, however, does not indicate integration of the information. It thus remains unresolved at what latency an integrated audio-visual neural representation emerges from the audio-visual interactions. To dissociate audio-visual (AV) integration from mere AV interaction in MEG/EEG signals, we designed a paradigm in which cross-modal integration was required both for the subject to perform a discrimination task and for the compared brain responses to differ. Four AV stimuli were created using two face images (V1 and V2) and two acoustic syllables /pa/ and /ka/ (A1 and A2). The two standard combinations A1+V1 and A2+V2 occurred with a probability of 0.4 each, while the two deviants, A1+V2 and A2+V1, occurred with a probability of 0.05 each. Additionally, a third image (V3) and the syllable /la/ (A3) occurred at random together either with A1 or A2 and either V1 or V2, respectively, with probability of 0.025 for each combination. In the first, bimodal condition, the deviant AV pairs served as targets in a GO/NO-GO task. In the second condition, the unimodal deviants (A3 and V3) served as targets in a unimodal GO/NO-GO task. Evoked responses were recorded with simultaneous whole-scalp 122-channel MEG and 64-channel EEG. Comparison of the average of the two standards with the average of the two deviants in the bimodal condition revealed a larger negative deflection to AV standards peaking at around 70 ms. The results indicated (p < 0.01) AV integration in the latency range of 60-105 ms (EEG, n = 10). The difference between bimodal standard and deviant responses in the unimodal task was not significant before 360 ms post stimulus, suggesting that strictly bimodal attention is required for early AV feature integration to take place at the cortical level. To our knowledge, this is the first unambiguous electrophysiological demonstration that early AV interactions can reflect true integration of auditory and visual features in the human brain. [1] Giard MH & Peronnet F, J. Cogn. Neurosci. 11:5, 473-490, 1999.

42	Visual-tactile sensory integration in man: a functional anatomy study using positron emission tomography
	Nicholas Stafford, M. Mehta, R. Banati, S. McGowan, J. Aggleton, M. Khwaja, P. Grasby, and C. Bench
	Combining information from different sensory streams is important for many cognitive and behavioural functions. We have extended our previous PET study of visual-tactile integration (1) by using (a) a modified version of the arc-circle test incorporating additional control conditions, (b) a more advanced PET camera with wider axial field of view, (c) reducing putative attentional load by including a learning procedure. Eight volunteers, 35-59 years, were tested using the modified arc-circle test, which required subjects to compare touched metal arcs with viewed circles and arcs on a computer screen in three different test conditions for which only instructions differed. Subjects were told to compare the right-hand arc with circles on the screen (touch-vision or TV), or the right-hand arc with the left-hand arc while looking at the screen (TT), or arcs to circles, both on the screen (VV).ð Subject responded using single words. A rest condition was included.ð Regional cerebral blood flow measures were made four times per condition, in a random order, using the H215O bolus method with the Seimens/CTI EXACT 966 PET camera.ð Data were analyzed using SPM99. We used P<0.001 (uncorrected) to signify activations in areas seen in our previous study; P<0.05 (corrected) was used for all other areas.ð The VV-TV comparison replicated our previous findings of bilateral visual cortex activation (BA18/19, P<0.001). TV-VV revealed activations in left BA40 (P<0.001) as reported previously and, in addition, activations in left BA3 (P<0.05) and right cerebellum (P<0.05) in the extended axial view. For TT-TV activations were present in right BA6 (P<0.05) and right BA1/3 (P<0.05). TV-TT revealed bilateral activations in BA18/19 (P<0.05) and left BA40 (P<0.05) more posterior to that in the TV-VV contrast. As previously (1), no amygdala activations were seen. Yet, in contrast, we did not identify activations in the anterior cingulate and DLPFC during cross-modal matching, perhaps due to the pre-scan training shown by improved and stable task performance. These findings suggest modulation of activity in primary sensory areas is specifically important in visual-tactile matching, possibly via selective attention as evidenced by activations seen in the parietal lobes. [1] Banati et al. (2000) Neuropsychologia 38:115-124.

43	Crossmodal Integration of Sight and Sound by the Human Brain
	Ingrid Olson, C.J. Gatenby and J.C. Gore
	Previous theories of crossmodal integration propose that sensory-specific areas are involved in representing crossmodal stimuli.ð Another theory proposes that only a central relay point is needed for crossmodal processing. To compare these two hypotheses, we used functional magnetic resonance imaging (fMRI) to identify the neural regions involved in audio-visual crossmodal integration of speech. Ten subjects were imaged while viewing a face in four different conditions: with speech and mouth movements synchronized, with speech and mouth movements desynchronized, during silent speech, or viewing a static face. Subtractions of the different sets of images showed that lip-reading, as defined by the silent speech - static face contrast, primarily activated Broca's Area and STS/STG. Synchronized audio-visual speech and desynchronized audio-visual speech activated similar areas, including Broca's Area, STG, and STS, suggesting that these areas are equally involved in the processing of unimodal and crossmodal stimuli. Regions activated more in the synchronized versus the desynchronized conditions were considered to be those involved in crossmodal integration. One dominant activation focus was found in the left claustrum/putamen. This study extends previous results, using other sensory combinations , and other tasks, indicating involvement of the claustrum in sensory integration. We hypothesize that the claustrum acts as a relay station for different sensory modalities while parts of the basal ganglia register the temporal co-occurrence of different events.

44	Fusing Vision and Touch Through Sensory Reweighting
	Kelvin Oie, Tim Kiemel and John J. Jeka
	The control of upright stance and postural orientation necessitates the integration of at least visual, somatosensory and vestibular inputs. These channels are held to provide both unique and redundant information relative for posture, enabling the resolution of potential ambiguities, as well as allowing the maintenance of posture when one or more sensory channels is not available. It has been suggested that in optimizing the control of stance these multiple inputs are re-weighted in a dynamic or adaptive manner. However, definitive support for sensory re-weighting is confounded by the possibility that the postural system may simultaneously adapt its control system parameters. By estimating the parameters of a third-order linear model that captures both the postural control system and its coupling to sensory information, we can assess whether sensory coupling and/or control parameters adapt as sensory conditions change. Utilizing a multisensory "moving room paradigm", we presented subjects with simultaneous, oscillatory visual and somatosensory stimuli at 0.2 Hz and 0.28Hz, respectively. Previous observations have shown that the postural response is strong with low-amplitude stimuli, dropping off as stimulus amplitude increases. Therefore, the stimuli used here manipulated relative stimulus amplitude in five conditions: (visual amplitude (mm): somatosensory amplitude (mm)) 4:16, 4:8, 4:4, 8:4 and 16:4. If the postural control system does not re-weight sensory inputs and/or change its control parameters, gain with respect to a given stimulus should not depend on the amplitude of either stimulus. However, if it does adapt to changing sensory conditions, we predict increasing gain with respect to low-amplitude stimuli as the amplitude of the other stimulus increases. Preliminary results showed systematic increases in gain to stimuli at constant amplitude (e.g., vision at 4 mm) when the other stimulus became large (e.g., touch at 8 and 16 mm), and vice versa. Furthermore, fitting model parameters for each condition suggested that the observed changes in gain were due primarily to changes in coupling strength parameters rather than to changes in control system parameters. These results suggest that the postural system may indeed re-weight sensory inputs under different conditions of stimulation. Supported by NIH grant R29 NS35070-01A2 (John J Jeka, PI).

45	Listening to Shapes: The Cross-Modal Equivalence of Vision and Audition in the Presentation of 2-D Shapes
	Eric C. Odgaard
	Spatial data are usually presented in some visual format, such as line graphs and two-dimensional shapes.ðIf the perceiver is either visually impaired or engaged in an eyes-busy task (such as surgery or piloting an aircraft), visual presentation of such information is impractical.ðThis study investigated the viability of displaying 2-D shapes (the border contours of U.S. states) in the auditory modality via pitch-time displays: auditory displays in which pitch represents vertical position and time represents horizontal position.ðCross-modal matching tasks revealed that perceivers can identify shapes at greater than chance levels; and that when errors in matching are made, they are based on similarities in features of the stimuli (rather than on random chance).ðMultidimensional Scaling Analyses further supported this conclusion, by revealing a stable set of salient features underlying the auditory presentation of the stimuli across several instruction sets. Comparison of the scaling solutions from the auditory presentations to those of a visual presentation revealed that, while the perceptual structure underlying the auditory presentation is stable, it is only partially equivalent to that of the more traditional visual modality.ðIn an applied sense, these results suggest that pitch-time displays may provide a viable alternative for the presentation of complex spatial data.ðIn a theoretical sense, these data would seem to further suggest that performance equivalence can occur between the modalities despite only limited perceptual equivalence between the stimuli.

46	Crossmodal interference between audition and touch
	Donna M. Lloyd, Steve Guest, Charles Spence, Natasha Merat and Francis P. McGlone
	We report a series of experiments designed to investigate crossmodal links in focused attention between audition and touch. Participants made speeded two-choice discrimination responses concerning the elevation (up vs. down) or roughness (rough vs. smooth) of targets presented in one modality, while trying to ignore irrelevant distractors presented simultaneously in the other sensory modality. The distractors were either congruent (i.e., an upper auditory target together with an upper tactile distractor) or incongruent (i.e., a rough tactile target surface together with the sound of a smooth surface) with the target. Robust compatibility effects were reported (i.e., responding was faster and more accurate when target and distractor were congruent than when they were incongruent) for targets in both modalities under a majority of conditions. These results add to a growing body of research demonstrating extensive crossmodal links between audition and touch in the control of attention.

47	Audio-visual sensory interaction in film perception
	Nicola Phillips
	Sensory interaction and integration in film perception was explored through a preliminary experimental study of the effects of concurrent musical listening and film viewing on perception of visual 'pulse'* in moving images.ðThis was grounded in two assumptions: that the contribution of audition and vision to integrated audio-visual percepts might be asymmetrical, reflecting the adaptation of these sensory systems for transmission of complementary information, and that the differing transduction times and temporal acuity of audition and vision might result in a processing bias towards auditorily-conveyed temporal information in filmic audio-visual perception. The experiment tested the hypothesis that perception of visual pulse would be modified by concurrent musical listening.ðParticipants viewed seven naturalistic film excerpts, each presented both without music and with various musical accompaniments.ðDuring viewing, participants tapped the perceived underlying visual 'pulse' of the moving images.ðComparison of response inter-onset intervals in 'with music' and 'visuals only' trials showed that the perceived underlying visual pulse of moving images was more rapid in the presence of music than in its absence.ðThe scope for musical listening to influence perception of underlying temporal structure of a moving image appeared to depend on stimulus ambiguity.*'pulse' is used in this sense to mean underlying pace of temporal articulation of a moving image, about moments of change or stasis.

48	Attending to Pain: Nociceptive Attention, Crossmodal Attention & Visual Dominance
	Francis P. McGlone, Charles Spence, Deborah E. Bentley, Paula D. Youell and Anthony K. P. Jones
	Mechanisms of selective attention serve to facilitate the selection of appropriate objects and events from amongst the cluttered and noisy scenes of everyday life.ð Many researchers have claimed to show that selective attention can also be directed to nociceptiveð (i.e., painful) stimuli? However, a number of alternative interpretations of these findings remain possible in terms of criterion shifts, task-switching, and spatial confounds. We examined whether attention to nociception could still be demonstrated when these alternatives were ruled out. Participants made speeded footpedal discrimination responses to an unpredictable sequence of nociceptive (pulsed CO2 laser) and visual stimuli presented to two locations on the left volar forearm. Attention was directed to one or the other modality by means of a symbolic cue at the start of each trial that predicted the likely modality for the upcoming target on the majority of trials. Significantly larger cuing effects were reported for visual stimuli than for nociceptive stimuli. The implication of these results for the multiple resource view of attention, and for the attentional account of visual dominance (Posner, Nissen & Klein, 1976) are discussed.

49	Both auditory and visual attention modulate activity in area MT+
	Rebecca Berman and Carol Colby
	The aim of this study was to determine whether attention to nonvisual stimuli can have an impact on activity in early visual cortex.ð We used the motion aftereffect (MAE) to assess visual motion processing in area MT+ during visual and auditory attention conditions. Perception of the MAE occurs after prolonged viewing of a unidirectional moving stimulus: when subjects subsequently view a stationary stimulus, it appears to be moving in the direction opposite to the adapting motion. Psychophysical studies have demonstrated that the MAE is diminished following attention to a central visual stimulus (Chaudhuri 1990), while functional imaging studies indicate that MAE perception is correlated with activity in area MT+ (Tootell et al. 1995). The present study brings together these psychophysical and FMRI findings to ask: Can attention to central visual or to auditory stimuli influence (1) perception of the MAE? and (2) activity in area MT+?We acquired psychophysical and FMRI data at 1.5 Tesla while subjects viewed moving and stationary stimuli in alternating blocks.ð Our FMRI measure of interest was the 'MAE signal': the time for activity in MT+ to decay back to baseline following different conditions of motion adaptation. In the first scan, we established each subject's MAE perception and associated MT+ activity in the absence of attentional demands by comparing reversing motion, which does not induce the MAE, with outward-only motion, which does induce the MAE.ð In the second scan, we asked whether this MAE signal was altered if the adapting stimulus was accompanied by a concurrent central visual task or auditory task.ð Attention to the central visual task caused the MAE signal to decay more rapidly than in the baseline (no-task) condition.ð Attention to the auditory task also caused a similar reduction of the MAE signal.ð Subjects' perception of the MAE paralleled the FMRI results, with diminished MAE perception following both visual and auditory attention conditions.ð These data indicate that attention to auditory stimuli can have an impact on early visual processing, and suggest that attention is not purely modality-specific.

 

50	Multisensory audio-somatosensory neural response interactions in early cortical processing: a high-density electrical mapping and behavioral study in humans
	Micah M. Murray, Beth A. Higgins, Deirdre M. Foxe, Kevin H. Knuth, Daniel C. Javitt, Charles E. Schroeder and John J. Foxe
	We examined the time course and scalp topography of auditory-somatosensory (A-S) neural response interactions in humans using high-density (128-channel) electrical mapping of somatosensory evoked potentials (SEPs) and auditory evoked potentials (AEPs). Tactile stimuli consisted of 15ms vibratory stimulation (~685Hz) of the fingertips of each hand. Auditory stimuli consisted of 15ms white noise bursts delivered to either of two speakers located next to each hand. Stimuli were presented either unimodally in isolation or bimodally in either conjunctive (same side) or disjunctive (different sides) simultaneous combinations. Subjects completed a simple reaction time task by making foot pedal responses to stimulus detection, regardless of whether stimuli were unimodal or bimodal, while maintaining central fixation. ERPs in response to bimodal stimuli were compared to the summed SEP and AEP from the corresponding unimodal conditions. These responses would be equivalent if neural responses to the unimodal stimuli were independent, whereas divergence is indicative of neural response interactions. In a previous study, we demonstrated A-S neural response interactions at ~50ms post-stimulus onset during passive conditions [1]. Similar interaction effects are seen in the current data set both when the bimodal stimuli were conjunctive combinations as well as when they were disjunctive combinations. Reaction times to bimodal stimuli were faster than those to unimodal stimuli, demonstrating a redundant target effect (RTE) for simultaneously presented A-S stimuli. Further, preliminary results indicate that the RTE is larger in the case of disjunctive versus conjunctive stimulus combinations. Probability summation could fully account for the RTE in each bimodal stimulus combination. Thus, although multisensory A-S neural response interactions occur early in cortical processing, they need not be invoked as a mechanism to describe the reaction time facilitation of the RTE. [1] Foxe J.J., Morocz I.A., Murray M.M., Higgins B.A., Javitt D.C., and Schroeder C.E. (2000) Multisensory Auditory-Somatosensory Interactions in Early Cortical Processing. Cognitive Brain Research, 10(1-2): 77-83.

51	Multisensory integration in the superior colliculus of the awake behaving primate
	Andrew Bell, B.D. Corneil, M.A. Meredith, J. Van Opstal, and D.P. Munoz
	The superior colliculus (SC), involved in the generation of saccades, also receives converging sensory inputs of different modalities and is a site of multisensory integration.ð We studied multisensory integration in awake, behaving monkeys.ð Single cell activity was recorded from the SC while monkeys performed a number of behavioral paradigms incorporating both visual and auditory stimuli.ð The paradigms were designed to investigate both the sensory response characteristics to multimodal stimuli and their influence on saccade related activity and resultant behavior.ð The paradigms included both fixation tasks as well as those which required monkeys to generate saccades to the stimuli.ð The influence of state of fixation was examined by presenting stimuli in both the gap condition (fixation point extinguished prior to stimulus presentation) and the overlap condition (fixation point visible during stimulus presentation). SC neurons responding to visual and/or auditory stimuli were examined.ð When the two stimuli were combined, we saw both response enhancement (response magnitude exceeded unimodal responses) and depression (magnitude decreased).ð Neurons which exhibited these interactions were located primarily in the deeper layers of the SC and often exhibited saccade related activity.ð These interactions were modulated by the spatial relationship between the two stimuli, the monkey's state of fixation (i.e., gap vs. overlap condition), as well as the parameters of the behavioral task. This study extends earlier findings on multisensory integration in awake, behaving primates and allows us to examine both its neural aspects as well as how these interactions affect the programming and characteristics of the ensuing saccades.ð Supported by HFSP Grant RG0174/1998-B.

52	Multisensory audio-visual interactions in early cortical processing: A high-density electrical mapping and behavioral study.
	Sophie Molholm, John J. Foxe, Micah M. Murray, Beth A. Higgins, Charles E. Schroeder, Daniel C. Javitt, and WalterðRitter
	Multisensory integration plays a pivotal role in human performance and perception.ð How and when information from separate sensory modalities comes together in the human brain is not well understood. Information regarding the timing and neural generators of multisensory intereactions is needed as a basis from which to understand how sensory integration operates.ð For example, are interactions driven by stimulus properties or higher order processes?ð Electrophysiological studies that have considered the timing of audio-visual interactions have shown variable onset times for such effects (e.g., de Gelder et al, 1999; Giard & Peronnet, 1999; Pourtois et al, 2000; Sams et al., 1991; Schrñger & Widmann, 1997).ð These differences are likely a result of stimulus, paradigm, and methodological differences.ð For example, while Giard and Peronnet (1999) have shown interaction effects as early as 40 ms over visual areas, Schrñger and Widmann (1997) did not show them until about 190 ms. The surprisingly early onset of Giard and PeronnetÌs (1999) interaction effects appear to belie current understanding ofð the activation time course of visual cortices, while the onset of Schrñger and WidmannÌs earliest effects seem late in light of what has been shown with intracranial investigations in animals.ð The present study re-examined the timing and topography of cortical audio-visual integration effects using high-density electrical mapping (128-channels). Single visual and auditory stimuli were presented either alone or together.ð Subjects were instructed to respond to all stimuli. ERPs to the visual alone and auditory alone stimulus conditions were summed (ÎsumÌ) and compared to the ERPs to the auditory and visual together (ÎpairÌ) condition.ð Divergence between the ÎpairÌ and ÎsumÌ ERPs indicated a neural response interaction.ð In the 200 ms post stimulus onset examined, interaction effects were present over visual areas at about 46-66 ms, corroborating the findings of Giard et al. (1999), and also at about 160-190 ms. Over auditory projection areas, significant interaction effects were observed at about 50-64, 110-130 and 175-200 ms.ðð The topography and timing of the earliest interactions are consistent with a system of areas in both visual and auditory association cortices that subserve multisensory audio-visual interactions early in the cortical processing hierarchy. In the behavioral data, reaction times were faster in the paired condition compared to the alone conditions, indicative of a redundant target effect. However, probability summation could fully account for this reaction time facilitation, providing no indication that neural response interactions facilitated reaction-time performance.

53	Auditory and Visual Objects
	David Van Valkenburg and Michael Kubovy
	Notions of objecthood have traditionally been cast in visuocentric terminology. As a result, theories of auditory and cross-modal perception have focused more on the differences between modalities than on the similarities. In this talk we will re-examine the concept of an object in a way which overcomes the traditional perspective. We propose a new, cross-modal conception of objecthood which focuses on the similarities between modalities instead of the differences. Further, we propose that the auditory system might consist of two parallel streams of processing (the 'what' and 'where' systems) in a manner analogous to current conceptions of the visual system. We suggest that the 'what' systems in each modality are concerned with objecthood. Finally, we present evidence for the hypothesis that the auditory 'where' system is in the service of the visuo-motor 'where' system.

54	Intersensory Redundancy Guides Perceptual Learning: Discrimination of Tempo in 3-Month-Olds
	Lorraine Bahrick, Robert Lickliter and Ross Flom
	Bahrick and Lickliter (2000) recently proposed an intersensory redundancy hypothesis, which holds that in early development information presented redundantly and in temporal synchrony across two sensory modalities selectively recruits attention and facilitates perceptual learning more effectively than does the same information presented unimodally.ð This hypothesis was supported by the finding that 5-month-old infants were able to differentiate between two complex, five-element rhythms when the rhythms were presented bimodally, but showed no evidence of differentiating the rhythms when they were presented unimodally (either visually or acoustically).ð The present study extended our test of the intersensory redundancy hypothesis by assessing discrimination of a new amodal property (tempo of action), in a group of younger infants (3-month-olds), using the same stimuli and procedures as the prior study. Thirty-two infants were habituated to films of a hammer tapping out a rhythmic sequence at a given tempo (55 or 120 beats per minute, counterbalanced), half under the bimodal (auditory and visual), and half under unimodal (visual) conditions.ðð Following habituation, infants received a change in tempo under their respective condition.ð Results replicated those of our prior study and demonstrated that infants showed significant discrimination of the change in tempo following bimodal habituation, t (15) = 2.92, p<.01, whereas they showed no evidence of discrimination following unimodal visual habituation.ð These findings support the intersensory redundancy hypothesis and suggest that bimodal stimulation selectively guides infant attention to redundant properties, and in turn, supports perceptual processing of those properties at the expense of others, giving a processing advantage to amodal information in early development.ðð It thus appears that when infants first learn to differentiate an amodal property, discrimination is facilitated by intersensory redundancy and attenuated under conditions of unimodal stimulation.

55	Tactile 'Capture' of Audition
	Anne Caclin, Salvador Soto-Faraco, Alan Kingstone and Charles Spence
	Previous research has demonstrated that the localization of auditory or tactile stimuli can be biased by the simultaneous presentation of a visual stimulus from a different spatial position. The authors investigated whether sound localization could also be affected by the presentation of spatially-displaced tactile stimuli, using a procedure designed to reveal any crossmodal perceptual interactions. Participants made left-right discrimination responses regarding the apparent location of sounds, which were presented in isolation, or together with tactile stimulation to the fingertips. The results demonstrate that the apparent location of a sound can be biased toward tactile stimulation when it is synchronous, but not when it is asynchronous, with the auditory event. Directing attention to the tactile modality did not increase the bias of sound localization toward synchronous tactile stimulation. These results provide the first unequivocal demonstration of the tactile 'capture' of audition.

56	Auditory-visual integration in human saccades elicited in a noisy environment
	Marc van Wanrooij, Brian Corneil, Doug Munoz, John van Opstal
	We report on behavioral ; experiments with five human subjects who made saccadic eye movements to a target that could be either visual-only (V), auditory-only (A), or auditory-visual (AV). In each trial, the target was hidden within an audio-visual background, and subjects were allowed up to 4.5 sec to identify the target as quickly as possible. To disambiguate the task in AV conditions, the spatial coordinates of the two stimuli always coincided. The target could be presented at either one of 24 possible locations, distributed over the 2D oculomotor range. To vary the strength of potential auditory-visual interactions, both the timing of the auditory target re. visual target onset, as well as its intensity with respect to the acoustic background noise (i.e. the S/N ratio), were systematically varied. All 17 different stimulus conditions (one V, 4 A, 12 AV conditions) were randomly interleaved, and for each subject, several recording sessions were performed on different days. In a control experiment, an additional AV condition was included in which the auditory target's location was unrelated to the visual target, thus creating an ambiguous bimodal stimulus condition. In all subjects tested, response accuracy of V saccades was high on average (as quantified by the response gain), but quite variable, and was typically endowed with long latencies. Often, subjects were unable to locate the target, resulting in idiosyncratic patterns of saccadic search sequences. The A-only saccades at the highest S/N ratios were slightly less accurate than V-saccades, but had much shorter latencies. Searching typically involved scanning along the elevation axis after the first saccade. As the S/N ratio of the A-stimuli was lowered, however, response latencies increased, and response accuracy for saccade elevation, and at the lowest S/N ratios also for saccade azimuth, systematically deteriorated. In the AV-condition, consistent auditory-visual interactions appeared in the first-saccade responses, that affected both the latency and the accuracy of the saccadic responses. At all S/N ratios tested, the saccades were as accurate, or better, as the V-only saccades, but at auditory latencies. Strongest effects were obtained for low S/N ratios, for which A-saccades were very inaccurate, but still have much shorter latencies than V-saccades. Search patterns were very rare in the AV condition, indicating that the auditory system helps the visuomotor system to follow the most likely route towards the target. These effects were absent in the AV control condition, and were less prominent in the condition in which the A-stimulus followed or preceded the V-stimulus by 100 ms. This result supports the notion that the AV interactions require spatial-temporal alignment.

57	Shifts of Intermodal Attention Induced by Endogenous Auditory Cues
	Kaiming G. Fu, Micah M. Murray, Beth A. Higgins, D.C. Javitt, C.E. Schroeder and John J. Foxe
	The brainÌs ability to selectively switch attention between modalities is necessary for efficient preferential processing of behaviorally relevant stimuli.ð We examined the mechanisms of this switching phenomenon in humans using high-density electrical mapping (64-channel).ð Endogenous auditory cues instructed subjects to perform either an auditory or visual discrimination (1.2s later) upon a compound auditory-visual stimulus.ð We focused on the 1.2s post-cue, pre-stimulus period to define the difference between auditory and visual attentional switch effects.ð Previous studies [1,2], using visual cues, have shown a difference in parietal-occipitalð ~10Hz activity during this timeframe.ð We proposed that this difference reflects a disengaged visual system in preparation for anticipated auditory input which is attentionally more relevant (see Foxe et al., 1998).ð The current study using endogenous auditory cues illustrated a similar effect during the same period.ð When cued to attend to the auditory modality and ignore the visual, the subjects exhibited significantly higher parietal-occipital ~10Hz amplitude, which began 400ms post-cue and lasted until the 2nd stimulus.ð These findings suggest that gating of the visual system occurs through similar mechanisms regardless of the modality of the instructive cue. References: [1] Foxe, J.J., Simpson, G.V., and Ahlfors, S.P. Cued shifts of intermodal attention: Parieto-occipital ~10 Hz activity reflects anticipatory state of visual attention mechanisms. Neuroreport, 1998, 9(17), 3929-3933. [2] Worden, M.S., Foxe, J.J., Wang, N., and Simpson, G.V. Anticipatory biasing of visuospatial attention indexed by retinotopically specific alpha-band EEG increases over occipital cortex. Journal of Neuroscience, 2000, 20: RC63 (1-6).

58	Somato-auditory convergence in auditory association cortex
	Taylor A. Johnston, K.G. Fu, T. McGinnis, J. Smiley, John J. Foxe, D.C. Javitt, C.E. Schroeder
	Recently-discovered somatosensory input to the caudo-medial (CM) auditory association cortex in macaques raises the possibility of multisensory integration in an area considered to have a unisensory processing function. Continuing investigation, has examined the qualities of the co-represented auditory and somatosensory inputs, as well as the anatomic substrates of somatosensory inputs into the region. Local patterns of transmembrane current flow and action potentials were indexed by sampling of current source density and multiunit activity profiles with linear array multielectrodes. Auditory stimuli were clicks (65db, 100 ms duration), tones and band-passed noise (65db, 100ms duration with 5ms on/off ramps). Somatosensory stimuli were provided by mild electrical stimulation of the median nerve at the wrist, and by light cutaneous stimuli applied to the hand as well as the rest of the body surface. Anatomical tracing utillized several flourescent compounds injected in small amounts through cannulae attached to multielectrodes; this facilitated precise placement of tracers. As reported earlier, characteristic frequency tuning in CM was broad relative to that in A1, and there was a preference for band-passed noise stimulation. Somatosensory receptive fields were large, and often bilateral. Findings thus far predict a complete body surface representation. Tracer experiments indicate several potential sources of the somatosensory input to CM, including Areas 5, 7 and SII/PV.

59	The Ventriloquist illusion in neglect patients
	Paul Bertelson, Francesco Pavani, Elisabetta Ladavas, Jean Vroomen and Béatrice de Gelder
	Can visual stimuli that go undetected, because they are presented in the extinguished region of neglect patientsà visual field, nevertheless shift in their direction the apparent location of simultaneous sounds (the well-known ventriloquist effect")? This issue was examined using a situation in which each trial involved the simultaneous presentation of a tone over loudspeakers, together with a bright square area on either the left, the right or both sides of  fixation . Participants were required to report the presence of squares, and indicate by hand pointing the apparent location of the tone. Five patients with left hemineglect consistently failed to detect the left square,  either presented alone or together with another square on the right. Nevertheless, on bimodal trials with a single undetected square to the left, their sound localization was significantly shifted in the direction of that undetected square.  By contrast,  on bimodal trials with either a single square on the right or a square on each side, their sound localization showed only small and non-significant shifts. This particular result might be due to a combination of low discrimination of lateral sound deviations with variable individual strategies triggered by conscious detection of the right square. The important finding is the crossmodal bias produced by the undetected left visual distractors. It provides a new example of implicit processing of inputs affected by unilateral visual neglect, and on the other hand is consistent with earlier demonstrations of the automaticity of crossmodal bias.

conference sponsor: Unilever conference contact: John J. Foxe website: s p r i n g f e l s .com