Wheat is a staple crop of Pakistan that covers almost 40% of the cultivated land and contributes almost 3% in the overall Gross Domestic Product (GDP) of Pakistan. However, due to increasing seasonal variation, it was observed that wheat is majorly affected by rust disease, particularly in rain-fed areas. Rust is considered the most harmful fungal disease for wheat, which can cause reductions of 20–30% in wheat yield. Its capability to spread rapidly over time has made its management most challenging, becoming a major threat to food security. In order to counter this threat, precise detection of wheat rust and its infection types is important for minimizing yield losses. For this purpose, we have proposed a framework for classifying wheat yellow rust infection types using machine learning techniques. First, an image dataset of different yellow rust infections was collected using mobile cameras. Six Gray Level Co-occurrence Matrix (GLCM) texture features and four Local Binary Patterns (LBP) texture features were extracted from grayscale images of the collected dataset. In order to classify wheat yellow rust disease into its three classes (healthy, resistant, and susceptible), Decision Tree, Random Forest, Light Gradient Boosting Machine (LightGBM), Extreme Gradient Boosting (XGBoost), and CatBoost were used with (i) GLCM, (ii) LBP, and (iii) combined GLCM-LBP texture features. The results indicate that CatBoost outperformed on GLCM texture features with an accuracy of 92.30%. This accuracy can be further improved by scaling up the dataset and applying deep learning models. The development of the proposed study could be useful for the agricultural community for the early detection of wheat yellow rust infection and assist in taking remedial measures to contain crop yield.