In recent years, virtual reality (VR) became more and more important in rehabilitation of upper extremities after stroke or other neurological diseases. Commercial gaming consoles, like the Microsoft Kinect, have been rapidly adopted in clinical settings. This paper presents the development of a rehabilitation game for the upper extremities with the Microsoft Kinect v2 sensor by simulating everyday situations. By using the Microsoft Kinect v2 sensor, the positions of the patients’ hands are tracked, which are further processed to emulate the movement of virtual hands in a virtual environment. A modified System Usability Scale (mSUS) was used to evaluate the usability of the rehabilitation game developed. Eleven stroke patients took part in a preliminary usability study where they had to complete the exercises of the presented rehabilitation game twice. The mSUS were filled by each patient and the standardized System Usability Scale by the supervising therapists. The rehabilitation game is implemented with two different exercises, where one is a pure training of the motoric functions where the task is to catch books falling out of a bookshelf. In contrast, the second exercise additionally trains the cognitive function of the patient as objects in a kitchen need to be moved and categorized by their names. The mean values of the mSUS filled out by the patients and SUS filled out by the therapists were 8.4 and 91.0, respectively. The results of the usability test show satisfying outcomes, as a SUS-score >68 indicates a good usability. These findings are confirmed by the patients’ feedback as they were very enthusiastic about this virtual rehabilitation scenario. To further evaluate the system developed, a usability study with 30 participants will be conducted.