Radio frequency fingerprint identification (RFFI) is an emerging device authentication technique, which exploits the hardware characteristics of the RF front-end as device identifiers. RFFI is implemented in the wireless receiver and acts to extract the transmitter impairments and then perform classification. The receiver hardware impairments will actually interfere with the feature extraction process, but its effect and mitigation have not been comprehensively studied. In this paper, we propose a receiver-agnostic RFFI system that is not sensitive to the changes in receiver characteristics; it is implemented by employing adversarial training to learn the receiver-independent features. Moreover, when there are multiple receivers, this functionality can perform collaborative inference to enhance classification accuracy. Finally, we show how it is possible to leverage fine-tuning for further improvement with fewer collected signals. To validate the approach, we have conducted extensive experimental evaluation by applying the approach to a LoRaWAN case study involving ten LoRa devices and 20 software-defined radio (SDR) receivers. The results show that receiver-agnostic training enables the trained neural network to become robust to changes in receiver characteristics. The collaborative inference improves classification accuracy by up to 20% beyond a single-receiver RFFI system and fine-tuning can bring a 40% improvement for under-performing receivers.