The rapid and accurate acquisition of rice growth variables using unmanned aerial system (UAS) is useful for assessing rice growth and variable fertilization in precision agriculture. In this study, rice plant height (PH), leaf area index (LAI), aboveground biomass (AGB), and nitrogen nutrient index (NNI) were obtained for different growth periods in field experiments with different nitrogen (N) treatments from 2019–2020. Known spectral indices derived from the visible and NIR images and key rice growth variables measured in the field at different growth periods were used to build a prediction model using the random forest (RF) algorithm. The results showed that the different N fertilizer applications resulted in significant differences in rice growth variables; the correlation coefficients of PH and LAI with visible-near infrared (V-NIR) images at different growth periods were larger than those with visible (V) images while the reverse was true for AGB and NNI. RF models for estimating key rice growth variables were established using V-NIR images and V images, and the results were validated with an R2 value greater than 0.8 for all growth stages. The accuracy of the RF model established from V images was slightly higher than that established from V-NIR images. The RF models were further tested using V images from 2019: R2 values of 0.75, 0.75, 0.72, and 0.68 and RMSE values of 11.68, 1.58, 3.74, and 0.13 were achieved for PH, LAI, AGB, and NNI, respectively, demonstrating that RGB UAS achieved the same performance as multispectral UAS for monitoring rice growth.