Combustion instability is a well-known problem in the combustion processes and closely linked to lower combustion efficiency and higher pollutant emissions. Therefore, it is important to monitor combustion stability for optimizing efficiency and maintaining furnace safety. However, it is difficult to establish a robust monitoring model with high precision through traditional data-driven methods, where prior knowledge of labeled data is required. This study proposes a novel approach for combustion stability monitoring through stacked sparse autoencoder based deep neural network. The proposed stacked sparse autoencoder is firstly utilized to extract flame representative features from the unlabeled images, and an improved loss function is used to enhance the training efficiency. The extracted features are then used to identify the classification label and stability index through clustering and statistical analysis. Classification and regression models incorporating the stacked sparse autoencoder are established for the qualitative and quantitative characterization of combustion stability. Experiments were carried out on a gas combustor to establish and evaluate the proposed models. It has been found that the classification model provides an F1-score of 0.99, whilst the R-squared of 0.98 is achieved through the regression model. Results obtained from the experiments demonstrated that the stacked sparse autoencoder model is capable of extracting flame representative features automatically without having manual interference. The results also show that the proposed model provides a higher prediction accuracy in comparison to the traditional data-driven methods and also demonstrates as a promising tool for monitoring the combustion stability accurately.