ABSTRACT The Internet of Things (IoT) requires sophisticated security due to heterogeneity and resource constraints. Current anomaly detection (AD) approaches address none of these challenges. Local AD models can account for device heterogeneity. However, existing approaches cannot run on constrained devices. This paper implements decentralized local AD models. Each model processes data from only one device. Simplifying the prediction task results in lightweight AD models. They provide an opportunity to address the resource constraints of devices. With less need for processing power, IoT devices can perform AD on their own. The novel approach improves the optimization metrics of detection performance, latency, bandwidth usage, privacy, and model complexity. Further optimization using model aggregation speeds up the creation of AD models. The evaluation uses the publicly available UNSW‐NB15 dataset. It shows that models can be simplified to run on IoT devices. Measurements with a local model on a Raspberry PI show only a slight increase in training and processing time compared with central remote processing on a significantly more powerful desktop PC. While the accuracy remains > 98%, the F1 score increases from 0.64 to 0.89 in the decentralized approach. The time for the creation of models is reduced by more than 90%.