We employed a novel cross-modal non-spatial inhibition of return (IOR) paradigm with fMRI to investigate whether object concept is organized by supramodal or modality-specific systems. A precue-neutral cue-target sequence was presented and participants were asked to discriminate whether the target was dog or cat. The precue and the target could be either picture or vocalization of a dog or cat. The neutral cue (bird) was always from the same modality as the precue. Behaviorally, for both visual and auditory targets, the main effect of cue validity was the only significant effect, p<0.01, with equivalent effects for within- and cross-modal IOR. Neurally, for visual targets, left inferior frontal gyrus and left medial temporal gyrus showed significantly higher neural activity in cued than uncued condition irrespective of the precue-target relationship, indicating that the two areas are involved in inhibiting a supramodal representation of previously attended object concept. For auditory targets, left lateral occipital gyrus and right postcentral gyrus showed significantly higher neural activity in uncued than cued condition irrespective of the cue-target relationship, indicating that the two areas are involved in creating a new supramodal representation when a novel object concept appears.