The upcoming Beyond 5G (B5G) and 6G networks are expected to provide enhanced capabilities such as ultra-high data rates, dense connectivity, and high scalability. It opens many possibilities for a new generation of services driven by Artificial Intelligence (AI) and billions of interconnected smart devices. However, with this expected massive upgrade, the privacy of people, organisations, and states is becoming a rising concern. The recent introduction of privacy laws and regulations for personal and non-personal data signals that global awareness is emerging in the current privacy landscape. Yet, many gaps need to be identified in the case of two data types. If not detected, then they can lead to significant privacy leakages and attacks that will affect billions of people and organisations who utilise B5G/6G. This survey is a comprehensive study of personal and non-personal data privacy in B5G/6G to identify the current progress and future directions to ensure data privacy. We provide a detailed comparison of the two data types and a set of related privacy goals for B5G/6G. Next, we bring data privacy issues with possible solutions. This article also provides future directions to preserve personal and non-personal data privacy in future networks.