Effective student group formation is crucial in higher education to foster collaborative learning environments. Grouping students by academic disciplines enhances peer-to-peer interactions and facilitates in-depth discussions on specialized topics. However, due to classroom space and resource constraints, it is challenging to accommodate all students from similar disciplines in one class. This necessitates a grouping method that can ensure a balanced distribution of students across available groups. Traditional K-means clustering, commonly used for this purpose, often results in inconsistent group sizes and fails to guarantee a balanced distribution of group members. Hard balanced clustering, which strictly enforces precise size limits on each cluster, offers a promising alternative for organizing balanced student sections to optimize classroom utilization. Nonetheless, most hard balanced clustering methods are limited in feature learning capability, which can lead to the overlooking of significant data patterns and result in ineffective clustering. To address this limitation, this paper introduces a new unsupervised model, Deep Hard Balanced Clustering (DHBC), which integrates hard balanced clustering with a deep learning framework to enhance feature learning. DHBC incorporates a balanced clustering mechanism within the optimization process of an Autoencoder architecture. It enhances the generated latent space representation by introducing a joint loss function that combines reconstruction and balanced clustering objectives, ensuring the embedded representation supports a balanced distribution of students. The model optimizes balanced clustering centroids during training. Comparative experiments conducted on real-world student enrollment datasets, evaluated by WCSS scores, demonstrate DHBC’s superiority in creating more cohesive and balanced student groups compared to state-of-the-art methods.