Granular materials undergo particle crushing under the long-term load, which changes the particle size distribution (PSD) of the material, and then affects its mechanical properties, so the problem of its long-term stability issues is outstanding. Previous research indicated that the crushing characteristics of coral sand particles have an effect on the creep under constant load. In order to reveal the response law of particle crushing over time during the creep process of coral sand, a series of one-dimensional confined compression tests under different stresses were carried out on coral sand taken from an area near an island and reef in the South China Sea. Creep behavior, particle fractal behavior, and particle crushing evolution during the creep process of coral sand were studied. The test results show that the creep of coral sand under different stress conditions exhibits significant non-linear decay creep characteristics. The power function can be used to mathematically describe the strain–time curve of coral sand. Coral sands with different initial distributions show more stringent self-similarity under the normal stress levels, and their fractal behavior was in the development stage. At higher stress levels, they show good self-similarity and significant fractal behavior. Under the same stress conditions, the crushing amount and fractal dimension of coral sand are larger than that of quartz sand. The process before the coral sand creep reaches stability was closely related to the amount of particle crushing. The particle crushing PSD development has a fractal trend and can be described by a gradually increasing fractal dimension and the relative particle crushing rate also shows a non-linear attenuation characteristic with time.