A seamless convergence between the connected TV and Virtual Reality (VR) ecosystems can open to the door to new fascinating scenarios. This paper initially introduces three EU H2020 projects (ImmersiaTV, ImAc and VRTogether) addressing relevant and complementary aspects towards achieving this integration. Next, an overview of the VRTogether project is provided. This project aims at enabling truly social VR experiences between remote users in domestic scenarios, based on photorealistic immersive content and on TV-related scenarios. Although the project is still in an early stage, its key objectives and requirements derived via user-centric activities are briefly introduced. Finally, the envisioned use cases and planned evaluations, together with the infrastructure and component architecture being deployed to support these scenarios, are also presented.