Constant exposures to fixed lags between keypresses and sensory stimulus affect subsequent temporal order judgments between keypresses and sensory stimulus in the same modality. This indicates that lag adaptation would recalibrate sensory-motor temporal relationship, and consequently shifts the point of subjective simultaneity (PSS). In this study, we examined the conditions for the transfer of the sensory-motor recalibration in temporal order judgment across modalities. In each trial, participants judged the temporal order between voluntary keypresses and the adapted or unadapted modality stimulus after a cue stimulus which urged them to voluntarily press a key. When participants judged the temporal order after the cue was presented in either of the modalities randomly, there was no significant shift in PSS. However, when participants judged the temporal order after the cue was presented to a fixed modality , the recalibration transferred across modalities although the PSS shift was smaller than the shift in the case in which the adaptation and test was restricted to a single sensory modality. These results suggest that keeping the attention toward one modality to trigger voluntary action is necessary condition for the transfer of the sensory-motor recalibration in temporal order judgment across modalities.