The implementation of the Authentication and Key Agreement (AKA) protocol in the Internet of Drones (IoD) is crucial for enhancing the security and reliability of information transmission. However, almost all existing authentication protocols between drones and Ground Station (GS) may suffer from several attacks due to capture attacks. In addition, the authentication between drones requires the participation of GS, which not only increases the amount of computation and transmission but also faces challenges such as impersonation attacks, lack of privacy protection, and perfect forward security. Therefore, we propose a secure and lightweight drone-to-GS (D2G) and D2D AKA protocol with perfect forward secrecy for IoD. Our protocol integrates physical unclonable functions (PUF) symmetrically into GS and drones to protect secret information against capture attacks while ensuring that GS does not store secret information related to drones. Furthermore, the proposed protocol enables direct mutual authentication between drones in a symmetrical manner without GS involvement, improving security and efficiency, particularly in scenarios where drones must collaborate without GS connectivity. Formal security proof using the random oracle model confirms the protocol’s resilience against various attacks. The performance analysis indicates that our scheme improves computational efficiency by an average of 39.44% compared to existing schemes that offer comparable security. Additionally, our approach incurs zero storage overhead during the GS authentication process. This protocol offers a secure and efficient solution for IoD, enhancing both security and scalability.