Histone deacetylases (HDACs) remove acetyl groups from acetylated lysine residues and have a large variety of substrates and interaction partners. Therefore, it is not surprising that HDACs are involved in many diseases. Most inhibitors of zinc-dependent HDACs (HDACis) including approved drugs contain a hydroxamate as a zinc-binding group (ZBG), which is by far the biggest contributor to affinity, while chemical variation of the residual molecule is exploited to create more or less selectivity against HDAC isozymes or other metalloproteins. Hydroxamates have a propensity for nonspecificity and have recently come under considerable suspicion because of potential mutagenicity. Therefore, there are significant concerns when applying hydroxamate-containing compounds as therapeutics in chronic diseases beyond oncology due to unwanted toxic side effects. In the last years, several alternative ZBGs have been developed, which can replace the critical hydroxamate group in HDACis, while preserving high potency. Moreover, these compounds can be developed into highly selective inhibitors. This review aims at providing an overview of the progress in the field of non-hydroxamic HDACis in the time period from 2015 to present. Formally, ZBGs are clustered according to their binding mode and structural similarity to provide qualitative assessments and predictions based on available structural information.