The agribusiness represents a significant portion of the global economy. In Brazil, agribusiness has a significant share of the country’s economy and represented 21.6% of GDP in 2017. To increase productivity, proper management of a crop, including pest control, is of vital importance. Annually, plant pests cause losses of 20% to 40% of production. For this reason, it is important to monitor the level of defoliation to take preventive actions. Therefore, in this work an automatic methodology is proposed using Convolutional Neural Networks, to detect the level of defoliation from leaf images in the soybean crop. In addition to detecting the presence of defoliation, the proposed methodology also provides the affected regions of the leaf through the segmentation of the image. Experimental results showed 83% accuracy using the proposed methodology versus 60% of SegNet CNN. The results are promising considering that the images were captured in the field, which presents challenges such as lighting, stages of development, scale, among others.