The noncommutative spacetimes associated to the $\kappa$-Poincar\'e relativistic symmetries and their "non-relativistic" (Galilei) and "ultra-relativistic" (Carroll) limits are indistinguishable, since their coordinates satisfy the same algebra. In this work, we show that the three quantum kinematical models can be differentiated when looking at the associated spaces of time-like worldlines. Specifically, we construct the noncommutative spaces of time-like geodesics with $\kappa$-Galilei and $\kappa$-Carroll symmetries as contractions of the corresponding $\kappa$-Poincar\'e space and we show that these three spaces are defined by different algebras. In particular, the $\kappa$-Galilei space of worldlines resembles the so-called Euclidean Snyder model, while the $\kappa$-Carroll space turns out to be commutative. Furthermore, we identify the map between quantum spaces of geodesics and the corresponding noncommutative spacetimes, which requires to extend the space of geodesics by adding the noncommutative time coordinate.