We present a broad class of semi-parametric models for time series of random sums of positive variables. Our methodology allows the number of terms inside the sum to be time-varying and is therefore well suited to many examples encountered in the natural sciences. We study the stability properties of the models and provide a valid statistical inference procedure to estimate the model parameters. It is shown that the proposed quasi-maximum likelihood estimator is consistent and asymptotically normally distributed. This work is complemented by simulation results and applied to annual growth rate time series of white spruce (Picea glauca) trees from a few dozen sites in Quebec spanning 41 years, including one major spruce budworm (Choristoneura fumiferana) outbreak from around 1968 to 1991. We found significant growth reductions due to budworm-induced by defoliation up to two years in the past. Our results also revealed positive effects of maximum temperature, precipitation and the climate moisture index in the summer, as well as negative effects of the climate moisture index in the spring and the maximum temperature in the previous summer. However, considering the interaction of climate and defoliation on growth did not improve the model's performance on this dataset. This study represent a major advances and our result represent an useful tool in the understanding of the combined effects of climate and insect defoliation on tree growth in the face of climate change, where the frequency and the severity of outbreaks, as well as an increase of temperature is expected.