Learning abstract representations from perceptual stimuli is a natural task for humans, but a real challenge for AI systems. In the vast majority of cases, in fact, systems that have to deal with symbol manipulation, like in reasoning or planning, do not need to also learn the symbols they operate on, but these are typically assumed to be given by a supervisor. Moreover, symbolic manipulation often implies compositional properties which are difficult to learn. In this paper, we consider the problem of learning symbolic representations that can be associated to abstract concepts, to be used in a variety of downstream tasks, and we analyze the many challenges related to this important problem, with a particular emphasis on paving the way towards composable symbolic representations learned by neural networks. We identify key properties for symbolic composable representations, such as non-ambiguity and purity, that suggest the need for different types of regularizations within the learning process, as well as new metrics for their evaluation.