Recent studies reported an involvement of motor/premotor brain areas during the perception of rhythm and beat in audition. However, little is known about the neural network of beat perception through non-auditory modalities. Here we used functional magnetic resonance imaging with auditory, vibro-tactile and visual rhythmic sequences in order to highlight the modality-specific areas involved in beat perception. We contrasted the brain activity changes in 20 healthy volunteers exposed to rhythmic sequences with a beat and control sequences without a beat. Results showed a recruitment of premotor cortex, supplementary motor area (SMA) and basal ganglia during beat sequences compared to rest in all three modalities. Beat sequences compared to no-beat sequences elicited the selective recruitment of the putamen and the SMA in the three sensory modalities, although to a lesser extent in vision. We conclude that the putamen and the SMA play a role in the prediction ability of a regular temporal pattern, although this seems less pronounced in vision compared to audition and to the vibro-tactile sensory modality.