Increasingly higher education programs are made open and flexible to face challenges demanded by societal changes. Challenge-based learning (CBL) is perceived as an educational concept shaping these open and flexible programs. However, CBL as a field of research is still in its infancy. The present study searches for all-embracing commonalities of CBL in engineering education. We propose an evaluative framework that both includes commonalities and allows for variety in CBL characteristics between study components. This framework, labelled CBL-compass, serves as a methodological approach for educational staff and researchers to visualise the local colour of CBL in higher education institutions. With this study we aim to advance the field by contributing to a conceptual basis in flexibility in CBL. Our research question was: How can we assess the variety of CBL implementations in engineering education experiments? This question was answered by an evaluative case study. First, existing literature on CBL was scoped. The characteristics following from this review were perceived as dimensions, each with associated indicators. Empirical data were collected from an evaluation of six CBL experiments. The variety of scores on the CBL-compass gave an impression of how teachers implemented CBL in their course or project and can thus be used as an evaluation mechanism to improve this implementation. Filling in the CBL-compass triggered reflection among teachers about their course and CBL. The added value of the CBL-compass is the attention for, amongst others assessment or teacher skills and support, which are important for the overall quality of study components.