Representing the position of the objects independently from our own position is a fundamental cognitive ability. Here we investigated whether this ability depends on visual experience. Congenitally blind, late blind and blindfolded sighted participants haptically learnt a room-sized regularly shaped array of objects, and their spatial memory was tested to determine which spatial reference frame was used. Crucially, the use of an object-based reference frame requires representing the regular structure of the array. We found that blindfolded sighted and late blind participants, that is those with visual experience, showed a preferential use of the object-based or 'allocentric' reference frame. On the contrary, congenitally blind participants preferred a self-based, or egocentric, reference frame. This suggests that, due to its developmental effect on the multisensory brain areas involved in spatial cognition, visual experience is necessary to develop a preference for an object-based, allocentric reference frame.