As a key technology to increase the system capacity in 5th generation (5G) mobile communications systems, ultra-dense networks (UDN) is proposed by deploying high-density wireless access points in hot spots. To mitigate the serious inter-cell interference(ICI) arising in UDN, we propose an adaptive clustering scheme as the basis for coordinated multipoint transmission and reception (CoMP), which has been proven to effectively eliminate interference. Two machine learning algorithms, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) and Particle Swarm Optimization (PSO), are introduced into the design of clustering scheme. Simulation results have shown that the proposed scheme can achieve a higher system throughput compared with modified K-means scheme. Furthermore, to be consistent with the concept of green communication, geographically isolated points could be identified and processed to save communicate resources with the proposed scheme.