AbstractIn this research, modified castor oil (MCO)‐based biocomposite has been synthesized and its structure is strengthened with waste polyethylene (PE) reinforcement. Both the petrochemical raw material used is reduced by 12 wt% and a new environmentally friendly biocomposite is produced using waste PE. Considering some thermophysical properties of the obtained biocomposite, experimental working conditions, and composition ratios have been optimized with response surface methodology (RSM). The chemical bond structure of the biocomposite has been investigated by Fourier transform infrared spectrophotometer, thermal decomposition behavior by thermogravimetric analysis, and surface morphology by scanning electron microscopy. According to the results obtained, the density and hardness of the biocomposite synthesized by the addition of MCO to unsaturated polyester (UP) decreases, and its thermal conductivity and thermal stability increase. The thermal decomposition kinetics of the biocomposite is also modeled with the newly improved hyperbolic function equation. The relationship between conversion rate and the temperature has been determined by the new model with a high correlation coefficient (R2 = 0.9985) and low‐error functions (SST = 0.0096, RMSE = 0.0285, χ2 = 0.0037). Effective and efficient use of MCO, UP, methyl ethyl ketone peroxide, and cobalt octoate in the production process has provided an economical and steady the biocomposite. Evaluation of experimental data with both RSM and artificial neural networks raises the reliability of the model results.