It is commonly understood that among the intermetallic phases used for permanent magnets, practically none can fully realize its potential based on the intrinsic magnetic properties. We discuss different reasons leading to this limitation, known as the Brown paradox, and discuss some possible ways of overcoming it. We look into Dy- and Tb-less grain boundary diffused magnets and the scope for the complete elimination of the heavy rare earth elements. Then we compare the intrinsic magnetic properties of (Nd1-xCex)2(Fe1-yCoy)14B single crystals with the extrinsic characteristics of sintered and hot compacted magnets, so-called rare earth balance magnets, made from the very same alloys. Finally, assessing RE-free materials, our results obtained on Mn- and Co-based RE-free single crystals are compared with the hard magnetic properties of Mn-based permanent magnets.