Heavy reliance on Cronbach’s alpha has been standard practice in many validation studies. However, there seem to be two misconceptions about the interpretation of alpha. First, alpha is mistakenly considered as an indication of unidimensionality and second, that the higher the value of alpha the better. The aim of this study is to clarify these misconceptions with the use of real data from the educational setting. Results showed that high alpha values can be obtained in multidimensional scales or tests given a sufficient number of items. Therefore, alpha cannot be an indication of unidimensionality. At the same time, after a certain point, higher values of alpha do not necessarily mean higher reliability and better quality scales or tests. In fact very high values of alpha could be an indication of lengthy scales, parallel items or a narrow coverage of the construct under consideration. Researchers are advised to apply caution when reporting alpha.