Predicting an upcoming event in time or modality leads to expectancy-based benefits for the predicted time point or modality and to costs for the unpredicted ones. Yet, it is not clear how expectancy to an event in time interacts with expectancy of a particular modality. In our study, participants had to perform a discrimination task (single or double pulse) on visual or tactile targets. An auditory cue (75% of validity) was used to increase expectancy in one or the other modality. We also manipulated the time point (1s or 3s after the cue) in which the stimulus appeared in a probabilistic way. We found that increasing the temporal expectation reduced the reaction time of valid targets in both, vision and touch, but when the target was presented in the unexpected modality (invalid cue) time predictability induced a cost in reaction time. Therefore the benefit of time expectancy is converted into a cost if the stimulus is presented in the unattended modality, suggesting cross-modal competition in the temporal domain.