Previous neuroimaging studies identified the fusiform face area (FFA) in the ventral visual cortex as a region specialized in the visual processing of faces. Here, we tested whether the FFA was recruited as well when a prosthesis substituting vision by audition (PSVA) was used. Using functional magnetic resonance imaging in blindfolded volunteers, we compared the brain activation patterns during the categorization of faces, man-made objects and meaningless images with the PSVA. Face-related activation foci were found in occipito-temporal brain areas, including the fusiform gyrus in and around the FFA. Man-made objects activated a larger occipito-temporal network including the same coordinates as faces, whereas meaningless images did not recruit any visual brain area. We conclude that perception of different visual categories provided by sensory substitution involves the same brain structures as direct vision, indicating a specific functional organization of the ventral visual stream to allow perception of faces or man-made objects independently of the sensory modality.