The vision of Industrie 4.0 includes the automated reduction of anomalies in flexibly combined production machine groups up to a zero-failure ideal. Algorithmic real-time detection of production anomalies may build the basis for machine self-diagnosis and self-repair during production. Several real-time anomaly detection algorithms appeared in recent years. However, different algorithms applied to the same data may result in contradictory detections. Thus, real-time anomaly detection in Industrie 4.0 machine groups may require a benchmark ranking for algorithms to increase detection results’ reliability. This paper makes a qualitative research contribution based on ten expert interviews to find design principles for such a benchmark ranking. The experts were interviewed on three categories, namely timeliness, thresholds and qualitative classification. The study’s results can be used as groundwork for a prototypical implementation of a benchmark.