Graph theory-based channel modeling is an efficient approach to simulate multipath propagation taking into account the reverberation of electromagnetic waves. In this contribution, without modifying the modeling framework, we proposed a semi-deterministic graph modeling approach by associating the scatterers with realistic environment objects, and by calculating the coefficients of the propagation paths on the base of a proven diffuse scattering (DS) theory. An indoor small office scenario is adopted to illustrate the procedure of the proposed method, and the influence of DS on propagation at millimeter wave frequencies is investigated, where 2 or 3 bounces of scattering is considered to be enough in the considered environment. The performance is thus evaluated by comparing the simulated power-delay-profiles (PDPs) and power-angular-spectra (PAS) with ray-tracing and real channel measurement data. The results illustrate that the proposed method can accurately predict both the decaying slopes and the diffuse tails of the PDPs. Furthermore, by combining the diffuse components generated by graph modeling with the specular component generated by ray-tracing, a good agreement between simulated and measured PAS can be observed.