The increasing interest in exploring robust transmitter authentication methods to enhance digital communication security has led to the emergence of physical layer authentication, a technique leveraging the intrinsic characteristics of a device to generate its fingerprints. This research investigates the feasibility of employing RF fingerprinting, as a physical layer authentication technique, in combination with deep learning algorithms to achieve transmitter device authentication. It proposes a transform technique called density trace plot (DTP) to efficiently exploit device-identifiable fingerprints resulting from physical layer imperfections. Notably, DTP operates directly on the raw received in-phase/quadrature signal, requiring no demodulation or decoding. However, authentication relying solely on deep learning is susceptible to authentication attacks from malicious adversaries. Given that a trained deep learning classifier, or any other classification machine learning model in general, always tries to map the input into one of the known classes, it is not equipped with the capability to detect malicious input. In response to this vulnerability, this research takes an additional step by incorporating radio frequency (RF) fingerprinting for spoof detection. Specifically, the study assesses the effectiveness of various adversarial neural networks (ANNs), including the variational autoencoder (VAE) and the generative adversarial network (GAN), in detecting spoofing attempts. This evaluation encompasses both simulated investigations and experimental testing. This thesis goes on to propose a framework for achieving rogue transmitter detection and legitimate device authentication for chirp modulation, particularly for LoRa applications. Considering that LoRa utilizes chirp modulation, which differs from other popular modulation schemes, it is essential to validate the framework's feasibility in LoRa system separately. Specifically, the continuous wavelet transform (CWT) is identified as a suitable technique for extracting fingerprints from LoRa signals. Once again, CNNs and GANs are leveraged to detect spoofing and authenticate transmitters.