Virtual Reality (VR) applications are currently suffering several gaps to allow a full breakthrough in the consumer market. One aspect is the social isolation. Many VR experiences do not represent yourself or other users, making it lonely rather than a shared experience. VR applications that do offer a user representation mostly use artificial avatars. These applications may not be well suited for all communication use cases (e.g. at work or with your family). In this paper I present our current work, at TNO and in my PhD research, to create Social VR experiences where both the user and the environment are represented in photo-realistic quality. The aim is not only to allow users to experience VR together, but also to allow new ways of natural communication within VR, with an extended presence and immersion. My PhD research focuses on 3 aspects that are building a Social VR system: capture and processing, transmission and client composition. The main goal of my research is to move processing from the capture and orchestration into the cloud (and particularly into the edge network) to allow Social VR with a large set of users (100+) in one session and to support mobile end devices while maximizing the QoS and QoE.