Augmented reality (AR) is a virtual technology that integrates virtual information and objects into real environments, offering unprecedented possibilities in such fields such as architecture, education, and healthcare. Real-time communication and security protocols are critical to the successful deployment of AR applications to ensure user immersion, prevent motion sickness, and address security problems. This paper proposes a secure user-to-user (U2U) and user-to-infrastructure (U2I) authentication protocol suitable for edge computing-based AR environments. We also employ extended Chebyshev chaotic maps and physical unclonable functions to ensure security and efficiency during the authentication process. The proposed protocol initiates session keys after U2I authentication when an AR user enters the edge node area, facilitating secure U2U authentication for sharing data with nearby users. We conduct comprehensive studies of the security robustness of the proposed protocol using formal and informal analyses, including “Burrows–Abadi–Needham logic”, “Real-Or-Random model”, the “Scyther tool” and informal security analyses. Furthermore, we measure the performance of cryptographic primitives using the “Multiprecision Integer and Rational Arithmetic Cryptographic Library” Cryptographic SDK. We perform a comparative analysis of security features and functionality, and we conduct a computational and communication cost analysis. The results reveal that the proposed protocol can provide security and efficiency for edge computing-based AR environments, presenting the methods for seamless and secure real-time AR data exchanges for U2I and U2U communications.