Touching Bouba, Hearing Kiki. Image resolution and sound symbolism in visual-to-auditory sensory substitution.
Last modified: 2013-05-05
Abstract
The Bouba/Kiki effect involves non-arbitrary mapping between visual shape and speech sounds (Ramachandran & Hubbard, 2001). We presented two groups of participants with both tactile 'bouba and kiki's' and their sonified soundscapes. One group trained in the use of a visual-to-auditory sensory substitution device (SSD), and the other a control group that received no training. Participants in the trained condition learned to make associations between visual and tactile objects and their soundscapes using the SSD. In the test phases the objects were categorized by image resolution and temporal duration. Even though higher resolution images were possible (maximum theoretical resolution of 176x144 pixels), successful object discrimination was significant down to 8x8 pixels for both tactile-to-auditory and visual-to-auditory conditions. Also, for duration, a highly significant difference favouring the 'short' duration category was found for the tactile-to-auditory condition.
Next, participants were tested with the Bouba/Kiki stimuli. Each group was tested with the tactile shape test. Participants in each condition primarily chose the expected tactile shape (88% trained condition; 83% control), consistent with that chosen visually. Next each group listened to SSD soundscape versions of the stimuli. Here only 41% of the naïve control participants chose the expected soundscape. In contrast, 76.5% of the trained participants selected the expected soundscape that was consistent with that chosen visually (all selected the expected visual-verbal association).
The results show that amongst naive users of SSD’s shape discrimination can be made using very basic object features and non-arbitrary cross-modal mappings are apparent after just a basic training regime.
Next, participants were tested with the Bouba/Kiki stimuli. Each group was tested with the tactile shape test. Participants in each condition primarily chose the expected tactile shape (88% trained condition; 83% control), consistent with that chosen visually. Next each group listened to SSD soundscape versions of the stimuli. Here only 41% of the naïve control participants chose the expected soundscape. In contrast, 76.5% of the trained participants selected the expected soundscape that was consistent with that chosen visually (all selected the expected visual-verbal association).
The results show that amongst naive users of SSD’s shape discrimination can be made using very basic object features and non-arbitrary cross-modal mappings are apparent after just a basic training regime.
Keywords
sensory substitution;perceptual learning;sound symbolism
References
Ramachandran, VS & Hubbard, EM (2001). "Synaesthesia: A window into perception, thought and language". Journal of Consciousness Studies 8 (12): 3–34.