We studied the effects of small, < 20 mum, Te inclusions on the energy resolution of CdZnTe gamma-ray detectors using a highly collimated X-ray beam and gamma-rays, and modeled them via a simplified geometrical approach. Previous reports demonstrated that Te inclusions of about a few microns in diameter degraded the charge-transport properties and uniformity of CdZnTe detectors. The goal of this work was to understand the extent to which randomly distributed Te-rich inclusions affect the energy resolution of CZT detectors, and to define new steps to overcome their deleterious effects. We used a phenomenological model, which depends on several adjustable parameters, to reproduce the experimentally measured effects of inclusions on energy resolution. We also were able to bound the materials-related problem and predict the enhancement in performance expected by reducing the size and number of Te inclusions within the crystals.