The growing popularity of eXtended Reality (XR) is being driven by technological advancements and the demand for advanced immersive digital experiences, including the vision around the metaverse. Within the XR realm, 360-degree immersive video streaming is essential for Virtual Reality (VR) adventures and experiences. The use of E2E encryption for content delivery in 360-VR streaming poses challenges for network operators, making it difficult to manage their networks and assess potential Quality of Experience (QoE) impairments, specifically in 5G and beyond networks. Therefore, we propose a Machine Learning (ML) approach for inferring 360-VR video QoE metrics from network-level encrypted traffic. Our solution uses packet-level information for feature engineering, which serves as input for the ML model to predict target QoE estimators. We evaluate our solution using real 4G and 5G drive test traces with encrypted VR traffic using HTTPS and QUIC protocols. The experimental results show that the trained ML model yields reasonable accuracy with minimal residual error in predicting target VR QoE for both HTTPS and QUIC. Network operators can use such a model to passively monitor the real-time QoE of encrypted VR video sessions and optimize network performance.