Open-ended questions are used to develop and capture students' cognitive engagement (CE) and are linked to an increase in language and aid in assessing aspects of student cognitive engagement. Assessing linguistic complexity and whether CE has occurred in open-ended questions is time-consuming. For this study, we present and compare Natural Language Processing and Machine Learning techniques for automatic detection of CE in open-ended questions using popular classifiers. Using data from a StoryQ curriculum using scaffolding and open questions, we assessed n= 28 students in three modules on machine learning practices(MLP). We developed a coding scheme adapted from two popular CE frameworks. The n = 840 CE coded responses were used to train three machine learning classifiers (i.e., support vector machine, random forest, and decision tree). The results showed that although each of the three classifiers scored better after tuning, SVM outperformed. However, the unbalanced dataset created challenges for the automatic classifier by misclassifying the higher engagement class. To better understand CE and students' literacy across the machine learning modules, we qualitatively analyzed students' responses by cross-recurrence quantification (CRQA) plots. We further compared students' CE levels to the relationship between the indices of the (CRQA), and completion percentage as predictors. Our results revealed that Completion Percentage, along with the indices Recurrence Rate (RR), Number of Recurrence Lines (NRLINE), and Average Line Length (L), were significantly related to the student's CE at both surface and deep levels. Based on the results from this study, scaffolded reading with engagement tasks, like open-ended questions, in teaching and learning on machine learning practices can produce higher levels of cognitive engagement and literacy complexity.