This paper focuses on emotion classification to investigate the relationship between rhythm metrics (Interval Measure (IM), and Pairwise Variability Index (PVI)), and emotions of speakers, all derived from the speech signal alone. The five considered emotions are “neutral,” “happiness,” “sadness,” “surprise,” and “questioning,” which are considered in the used corpus for the Modern Standard Arabic (MSA) dialect. Three classifiers were designed and tested on the King Abdulaziz City for Science and Technology Text to Speech Database (KTD) to classify the above five emotions. We found variations between the different emotion categories, especially with regard to sadness and questioning. A major conclusion is that ΔC, ΔV, and rPVI rhythm metrics can be used to classify emotions such as those investigated in this study. All three classifiers tested achieved a correctness rate above 62%.