How do people remember the location of objects? Location is always relative, and thus depends on a reference frame. There are two types of reference frames: egocentric (or observer-based) and allocentric (or environmental-based). Here we investigated the reference frame people used to remember object locations in a large room. We also examined whether the choice of a given reference frame is dictated by visual experience. Thus we tested congenitally blind, late blind, and sighted blindfolded participants. Objects were organized in a structured configuration and then explored one-by-one with participants walking back and forth from a single point. After the exploration of the locations, a spatial memory test was conducted. The memory test required participants to imagine being inside the array of objects, being oriented along a given heading, and then pointing towards the required object. Crucially the headings were either aligned to the allocentric structure of the configuration, that is rows and columns, or aligned to the egocentric route walked during the exploration of the objects. The spatial representation used by the participants can be revealed by better performance when the imagined heading in the test matches the spatial representation used. We found that participants with visual experience, that is late blind and blindfolded sighted, were better with headings aligned to the allocentric structure of the configuration. On the contrary, congenitally blind were more accurate with headings aligned to the egocentric walked routes. This suggests that visual experience during early development determines a preference for an allocentric frame of reference.