A 3D numerical model based on Lattice Boltzmann Method (LBM) has been developed to investigate the particle behavior of mixed-grain-size sediments under oscillatory flows and progressive waves. The model considers both particle-particle interaction and particle-fluid interaction with free surface flow and SGS turbulence model. By applying the present model, simulations of mixed-grain-size sediments with different influence factors (i.e., Shields parameters, periods, bottom layer thicknesses) under symmetric oscillatory flows as well as under progressive cnoidal wave are performed to understand the vertical sorting process of graded sands. The results of mixed sands under symmetric oscillatory flows show that the water particle semi-excursion is a key influence factor on the armoring effect. For the cnoidal wave, the vertical sorting proceeds until the wave crest passes by and it fully develops in a complete period. The concentration centroid of large particles becomes higher landward within 2 periods.