We examined that incongruent visual image of one’s hand impaired the tactile perception to the finger. Participants judged which one of the left hand fingers was tapped as quickly and correctly as possible, ignoring the light tap on the right hand finger. We used a mirror in order to manipulate the presence of visual image of the left hand. In the visual-image-presence condition, the mirror was placed between participants’ hands and thus the mirror image of the right hand was seen as if it was the image of the left hand. Therefore the visual image and the tactile stimulus were incongruent when the tapped fingers were different between both hands. The participants’ responses in the visual-image-presence condition were compared with the condition in which an opaque board was used instead of the mirror (the control condition). In the results, the incongruent visual image prolonged the response times and increased the error rates compared with the same situations in the control condition. Participants rarely made incorrect responses or had difficulty in responses when there was no incongruent visual image. These results indicated that vision strongly influenced the tactile perception even in the very easy tactile discrimination task.