The quest for atomically precise synthesis of colloidal semiconductor nanostructures has attracted increasing attention in recent years and remains a formidable challenge. Nevertheless, atomically precise clusters of semiconductors, known as magic-size clusters (MSCs), are readily accessible. Ultrathin one-dimensional nanowires and two-dimensional nanoplatelets and nanosheets can also be categorized as magic-size nanocrystals (MSNCs). Further, the magic-size growth regime has been recently extended into the size range of colloidal QDs (up to 3.5 nm). Nevertheless, the underlying reasons for the enhanced stability of magic-size nanostructures and their formation mechanisms remain obscure. In this Perspective, we address these intriguing questions by critically analyzing the currently available knowledge on the formation and stability of both MSCs and MSNCs (0D, 1D, and 2D). We conclude that research on magic-size colloidal nanostructures is still in its infancy, and many fundamental questions remain unanswered. Nonetheless, we identify several correlations between the formation of MSCs and 0D, 1D and 2D MSNSs. From our analysis, it appears that the "magic" originates from the complexity of a dynamic and multivariate system running under reaction control. Under conditions that impose a prohibitively high energy barrier for classical nucleation and growth, the reaction proceeds through a complex and dynamic potential landscape, searching for the pathway with the lowest energy barrier, thereby sequentially forming metastable products as it jumps from one local minimum to the next until it eventually becomes trapped into a minimum that is too deep with respect to the available thermal energy. The intricacies of this complex interplay between several synergistic and antagonistic processes are, however, not yet understood and should be further investigated by carefully designed experiments combining multiple complementary in situ characterization techniques.