AbstractBroken coal and rock masses are the major part of the goaf. The compaction characteristics of coal and rock masses and the breakage law of whose particles during compaction exert an important influence on various aspects including control of strata motion, prediction of surface subsidence, and backfill mining. In this paper, the triaxial compaction experiment on broken coal‐rock masses with different mixing ratios was carried out. The test results showed that with the increase of stresses, the strain of coal‐rock masses gradually rose while the porosity, bulking factor, and degree of compaction gradually declined. During the compaction of coal‐rock masses, the fitting curves of the strain, porosity, bulking factor, and degree of compaction with stresses of coal samples all appeared as a cubic function of stresses. The breakage behavior of coal particles underwent three stages: structure re‐arrangement and breakage of particles, particle breakage, and compression‐induced deformation of particles. With increasing stress, the crushing amount of particles gradually grew while the increase rate of the crushed particles gradually decreased and the larger the particle strength was, the lower the increase rate of the crushing amount. Additionally, in the compaction process of samples, particle breakage mainly appeared before the stress reached to 8 MPa while the coal and rock particles were hardly crushed after the stress was larger than 8 MPa. With increasing stresses, the particle size gradation of samples gradually became reasonable and the lower the particle strength of samples was, the more reasonable the particle size gradation of compacted samples. The particle size gradation of various compacted and crushed samples showed a favorable fractal characteristic. In the stage with a low stress, the value of fractal dimension D rapidly grew and the fractal dimensions D of various samples tended to be stabilized after the stress reached to a high level.