The traditional approach for analyzing the quality of arecanuts is based on their external appearance. However, using machine learning and deep learning techniques, automated classifications were performed. But the true quality can only be analyzed when the internal structure of the arecanut is examined. Therefore, we use the X-ray imaging technique to determine the internal quality of arecanuts. We prepared a novel dataset of arecanut X-ray images and used a YOLOv5 based deep learning architecture for classification. The present study employs an adaptive genetic algorithm based approach for hyperparameter optimization to enhance the mean average precision (mAP) using a light weight model generated using a ghost network and a feature pyramid network (FPN). We have achieved the highest mAP of 97.84% using our method with a lower model size of 15 MB. Our method has excelled in detecting the arecanut compared to cutting-edge object detection algorithms such as YOLOv3, YOLOv4, Detetron, YOLOv6, YOLOv8, and YOLOX. We also acknowledged the performance enhancement using the adaptive genetic algorithm on the Pascal VOC 2007 image dataset. Despite of significant computational requirements for executing genetic algorithms, we proved that genetic algorithms can boost mAP. Additionally, the methodology developed in this investigation produced multiple models with the best mAP featuring optimized hyperparameters. This methodical strategy is helpful for the design of an automatic, non-destructive, integrated X-ray image based classification system. This system has the potential to revolutionize the quality assessment of arecanuts by offering a more efficient evaluation method.