This study investigates the effect of tooth profile modification on the vibrations of a single stage spur planetary gearbox. A lumped mass nonlinear dynamic model of the gearbox is used [1] to study the effect of tooth profile modifications applied on the sun-planet and ring-planet meshes on vibration behavior. In order to avoid modification on ring gear, both tip and root reliefs are considered for sun and planet wheels. The static model for planetary gear system with tooth profile modification is validated through comparisons with a static finite element analysis of the whole system, performed using commercial software. In the present study, an optimization approach based on Genetic Algorithms (GA) is proposed to improve planetary gear dynamic performances toward vibration reduction. As proven by dynamic analyses, a genetic algorithm is an effective optimization tool to design reliable profile modifications for reducing the planetary gears vibration amplitude over a wide frequency range.