AbstractMessage Queuing Telemetry Transport (MQTT) is the most widely used protocol within the communication layer of the Internet of Things (IoT). Message brokers are a key component of the MQTT protocol and a single point of failure. Incompatibilities between different MQTT brokers or broker versions with their clients can cause critical failures and become a source of security risks. Thus, every MQTT broker change or update needs to be accompanied by an evaluation of the compatibility between the new and the previous broker. In this work, we develop an automated framework for compatibility evaluation of MQTT brokers, which can be easily generalized to other similar IoT components. We apply this framework to perform a comprehensive experiment conducted with 16 different versions of 6 popular MQTT brokers. We report inconsistencies in the behavior of different MQTT brokers and broker versions. Based on the experiment results, we calculate and provide a visualization of compatibility among the evaluated brokers in terms of their distance, which indicates the risk of incompatibilities when replacing a broker with another one. The calculation of distance measures can be adjusted by giving higher weights to important features. We use this method to show security‐related differences between the brokers.