This paper presents a mesoscale model to study the influence of rubber particles on the mechanical performance of crumb rubber mortar (CRM). The indirect tensile and flexural behaviors of CRM with different rubber replacement rates, shapes, and sizes were investigated. Rubber mortar is assumed to be a three-phase material composed of rubber aggregate, a mortar matrix, and an interface transition zone (ITZ). Numerical analysis showed that rubber content was the governing factor affecting the reduction rate of indirect tensile and flexural strength. The effect of the ITZ on the tensile strength of CRM was within one percent, which could be ignored. The influence of rubber particle size was investigated by analyzing CRM models containing five different rubber sizes from 0.86 mm to 7 mm. For each size, six different models with randomly distributed rubber particles were set up. CRM models presented a similar average strength even with different rubber particle sizes. However, the strength variation among the random models became higher when the rubber particle size increased. Numerical results also proved that treating rubber particles as pores in modeling led to negligible errors. Then, a prediction formula after considering the increase in air content is provided. Finally, the accuracy of numerical simulations was verified through a series of experimental studies.