Object recognition studies typically focus on shape rather than surface properties and have almost exclusively involved vision. Visual object recognition is impaired when surface properties, such as color, change between study and test, suggesting that these properties form part of the visual representation of the object (Nicholson & Humphrey, Perception 32:339-353, 2004). Cross-modal recognition of objects is hypothesized to rely on a modality-independent representation of shape but whether this representation also contains information about surface properties is unknown. Here, we manipulated the color and texture of objects to investigate their effect on visual and haptic within-modal and cross-modal object recognition (both visual-haptic and haptic-visual). In Experiment 1, changing the color schemes of objects impaired within-modal visual recognition but had no effect on either cross-modal condition or, unsurprisingly, on within-modal haptic recognition. In Experiment 2, we changed the texture schemes of objects. Changing the texture schemes impaired recognition in all conditions. This shows that surface properties may form part of haptic object representations, as previously shown for vision. The reduction in cross-modal recognition when textures changed suggested that the underlying modality-independent representation encodes surface properties. However, this result could also simply reflect a general disruptive effect of changes in modality-independent surface properties that was not present in Experiment 1. To address this, Experiment 3 examined the effect of texture and orientation changes. Because the modality-independent representation underlying cross-modal recognition is also view-independent (Lacey, Peters & Sathian, PLoS ONE, 2:e890, 2007), if texture changes reduced recognition but without affecting view-independence, this would indicate that the modality-independent representation also encodes surface properties. We found that cross-modal recognition was indeed impaired by changes in texture but not changes of orientation. We conclude that the modality-independent, view-independent object representation does contain information about modality-independent surface properties.