Abstract Malware remains a big threat to cyber security, calling for machine learning-based malware detection. Malware variations exhibit common behavioral patterns indicative of their source and intended use to enhance the existing framework’s usefulness. Here we present a novel model, i.e., African Vulture Optimization-based Decision Tree (AVO-DT) to increase the overall optimization. The datasets from Android apps and malware software train the AVO-DT model. After training, the datasets are pre-processed by removing training errors. The DT algorithm is used by the developed AVO model to carry out the detection procedure and predict malware activity. To detect malware activities and improve accuracy, such an AVO-DT model technique employs both static and dynamic methodologies. The other measurements on Android applications might be either malicious or benign. Here we also developed malware prevention and detection systems to address ambiguous search spaces in multidimensionality difficulties and resolve optimization challenges.