"Imaginary space" is a three-dimensional visual awareness that feels different from what you experience when you open your eyes in broad daylight. Imaginary spaces are experienced when you look "into" (as distinct from "at") a picture for instance. Empirical research suggests that imaginary spaces have a tight, coherent structure, that is very different from that of three-dimensional Euclidean space. This has to be due to some constraints on psychogenesis, that is the development of awareness. I focus on the topic of how, and where, the construction of such geometrical structures, that figure prominently in one's awareness, is implemented in the brain. My overall conclusion-with notable exceptions-is that present day science has no clue. I indicate some possibly rewarding directions of research.