A microstructure-based representative volume element (RVE) method is employed for simulating the mechanical properties of the alloy steel 42CrMo4, a typical ferrite–cementite steel with ferrite (α) and spheroid cementite (θ) phases. A series of 2D RVEs with ferrite matrix and globular cementite particles are generated according to various microstructures, among which the geometrically necessary dislocation (GND) accumulation at the α–θ interphase is introduced as a coating layer around the cementite particles. Satisfactory agreement between the simulation and experimental flow curves can be obtained when the ratio of GND layer thickness to particle diameter is 0.25. On this basis, the influences of some modeling parameters such as element type and boundary conditions, as well as some microstructure features like particle size and carbide band, on the mechanical properties are investigated. The results show that small particle size can increase the strength of the material. Carbide band leads to the microstructure inhomogeneity and early finish of uniform elongation, which is potential to cause the loss of ductility and premature failure. These adverse effects become more severe when more cementite particles remain in band or the cementite particles gather denser in carbide band.