Previous studies suggest that the brain network responsible for the recognition of facial expressions of emotion (FEEs) begins to emerge early in life. However, it has been unclear whether visual experience of faces is necessary for the development of this network. Here, we conducted both psychophysical and functional magnetic-resonance imaging (fMRI) experiments to test the hypothesis that the brain network underlying the recognition of FEEs is not dependent on visual experience of faces. Early-blind, late-blind and sighted subjects participated in the psychophysical experiment. Regardless of group, subjects haptically identified basic FEEs at above-chance levels, without any feedback training. In the subsequent fMRI experiment, the early-blind and sighted subjects haptically identified facemasks portraying three different FEEs and casts of three different shoe types. The sighted subjects also completed a visual task that compared the same stimuli. Within the brain regions activated by the visually-identified FEEs (relative to shoes), haptic identification of FEEs (relative to shoes) by the early-blind and sighted individuals activated the posterior middle temporal gyrus adjacent to the superior temporal sulcus, the inferior frontal gyrus, and the fusiform gyrus. Collectively, these results suggest that the brain network responsible for FEE recognition can develop without any visual experience of faces.