Background: Patients recovered from COVID-19 often suffer from the sequelae of the disease, which can hinder the patients’ activity in daily living. Early recognition of the patients at risk of prolonged hospitalization and impaired physical functioning is crucial for early intervention. We aim to identify the predictors of prolonged hospitalization and impaired activity in daily living in this study. Methods: COVID-19 patients hospitalized in a medical center were divided into two groups according to the Barthel index three months after discharge and the median length of hospital stay, respectively. Chi-square test and Mann–Whitney U test were performed to check the differences between the two groups in patient characteristics as well as hematology tests at the emergency department, the intensive care unit mobility scale (ICUMS), and the medical research council sum score (MRCSS). Logistic regression and the receiver operating characteristic curve analysis were further performed for the factors with significant differences between the two groups. Results: Both ICUMS and MRCSS showed significant differences between the groups. The ICUMS had an odds ratio of 0.61 and the MRCSS of 0.93 in predicting a Barthel index score less than 100 three months after discharge. The MRCSS had an odds ratio of 0.82 in predicting a prolonged length of hospital stay. Conclusion: Both ICUMS and MRCSS upon admission are predictive of a Barthel index score of less than 100 three months after discharge. On the other hand, only MRCSS has predictive value of a prolonged hospitalization.