In this paper we introduce the notion of the center of distances of a metric space, which is required for a generalization of the theorem by J. von Neumann about permutations of two sequences with the same set of cluster points in a compact metric space. Also, the introduced notion is used to study sets of subsums of some sequences of positive reals, as well for some impossibility proofs. We compute the center of distances of the Cantorval, which is the set of subsums of the sequence $\frac34, \frac12, \frac3{16}, \frac18, \ldots , \frac3{4^n}, \frac2{4^n}, \ldots$, and also for some related subsets of the reals.