Predictive maintenance attempts to prevent unscheduled downtime by scheduling maintenance before expected failures and/or breakdowns while maximally optimizing uptime. However, this is a non-trivial problem, requiring sufficient data analytics knowledge and labeled data, either to design supervised fault detection models, or to evaluate performance of unsupervised models. While today most companies collect data by adding sensors on their machinery, the majority of this data is unfortunately not labeled. Moreover, labeling requires expert knowledge and is very cumbersome. To solve this mismatch, we present an architecture that guides the experts so they only have to label a very small subset compared to today's standard labeling campaigns when designing predictive maintenance solutions. We use autoencoders to highlight potential anomalies and clustering approaches to group anomalies into (potential) failure types, and then present the resulting clustered anomalies to domain experts for labeling. This way, we enable domain experts to enrich routinely collected machine data with business intelligence via a user-friendly hybrid model, combining autoencoder models with labeling steps and supervised models. Ultimately, the labeled failure data allows creating better failure prediction models, which in turn enables more effective predictive maintenance. More specifically, our architecture opens up the data available within companies without cumbersome labeling tasks, so companies make maximum use of their data and expert knowledge to ultimately gain increased profit. Using our methodology, an average of 85 % of labeling gain is achieved when compared to standard labeling tasks.