Gaussian mixture probability hypothesis density (GM-PHD) filtering based on random finite set (RFS) is an effective method to deal with multi-target tracking (MTT). However, the traditional GM-PHD filter cannot form a continuous track in the tracking process, and it is easy to produce a large number of redundant invalid likelihood functions in a dense clutter environment, which reduces the computational efficiency and affects the update result of target probability hypothesis density, resulting in excessive tracking error. Therefore, based on the GM-PHD filter framework, the target state space is extended to a higher dimension. By adding a label set, each Gaussian component is assigned a label, and the label is merged in the pruning and merging step to increase the merging threshold to reduce the Gaussian component generated by dense clutter update, which reduces the computation in the next prediction and update. After pruning and merging, the Gaussian components are further clustered and optimized by threshold separation clustering, thus as to improve the tracking performance of the filter and finally realizing the accurate formation of multi-target tracks in a dense clutter environment. Simulation results show that the proposed algorithm can form a continuous and reliable track in dense clutter environment and has good tracking performance and computational efficiency.