On a non-spatial task, individuals respond more slowly when the stimulus and response are on different sides of space versus the same side of space (the Simon effect). Past Simon effect studies have shown that visual stimuli are represented based on external reference frames. In contrast, we have found that tactile stimuli in a Simon effect task are encoded based on a somatotopic reference frame (e.g. stimuli presented to the left hand are always encoded as left regardless of the hand’s location in space). In order to understand how visual stimuli are represented on or near the body, we presented individuals with a visual Simon effect experiment. Using foot pedals, participants responded to visual stimuli projected on, near, or far from the hands in three conditions – hands uncrossed, hand crossed, and no hands. In the no hands condition, visual stimuli were encoded based on an external reference frame. However, we find that as with tactile stimuli, visual stimuli presented on or near the hands (crossed condition) were encoded based on a somatotopic frame of reference. These novel results provide evidence that visual stimuli on or near the hands can be encoded based on body representations.