The molecular properties of CD8+ T cells that respond to SARS-CoV-2 infection are not fully known. Here, we report on the single-cell transcriptomes of >80,000 virus-reactive CD8+ T cells, obtained using a modified Antigen-Reactive T cell Enrichment (ARTE) assay, from 39 COVID-19 patients and 10 healthy subjects. COVID-19 patients segregated into two groups based on whether the dominant CD8+ T cell response to SARS-CoV-2 was 'exhausted' or not. SARS-CoV-2-reactive cells in the exhausted subset were increased in frequency and displayed lesser cytotoxicity and inflammatory features in COVID-19 patients with mild compared to severe illness. In contrast, SARS-CoV-2-reactive cells in the dominant non-exhausted subset from patients with severe disease showed enrichment of transcripts linked to co-stimulation, pro-survival NF-κB signaling, and anti-apoptotic pathways, suggesting the generation of robust CD8+ T cell memory responses in patients with severe COVID-19 illness. CD8+ T cells reactive to influenza and respiratory syncytial virus from healthy subjects displayed polyfunctional features and enhanced glycolysis. Cells with such features were largely absent in SARS-CoV-2-reactive cells from both COVID-19 patients and healthy controls non-exposed to SARS-CoV-2. Overall, our single-cell analysis revealed substantial diversity in the nature of CD8+ T cells responding to SARS-CoV-2.