The low number of students studying science, technology, engineering, and mathematics (STEM) has been a cause for concern for the past decades in the Western countries. Furthermore, the student body in STEM has narrowed since the number of female students in STEM fields decreases despite outreach activities. In order for the society to have enough appropriately educated work force, STEM attractiveness and increasing it is the focus of this concept paper. Although the lack of interest in STEM studies has been analyzed widely, we still have not been able to attract more (female) adolescents in STEM fields. Furthermore, previous research has shed light on the variety of STEM outreach activities, but the effects of these activities are not definitive. These concerns are common also in the Nordic countries, which has led to establishing an Erasmus+ Strategic Partnerships project 'Engineering Nordic Future' in 2018. One of the focus areas of the project pertains to the ways of attracting adolescents into STEM studies. As part of this project, the present literature review identifies the key features that contribute to the effectiveness of STEM outreach activities. The analysis focuses on K-12 activities targeted for (female) adolescents. The resulting model provides criteria for improving the effectiveness of current STEM outreach activities. The criteria may be applied to evaluate the impact of university-led STEM outreach activities and to guide related decision-making. 1 INTRODUCTION As we are currently in the midst of the fourth industrial revolution, which entails digitalization, the Internet of Things, smart factories, and other advanced technological processes and entities, our demand for expertise in those areas is ever increasing. Despite this, it appears that students are less attracted to science, technology, engineering, and mathematics (STEM) related majors in tertiary education. Furthermore, especially in the Nordic countries, the number of female students has decreased notably [1]. The careers within STEM majors are quite varied [1] - [2], but the number of students entering STEM studies is stagnating [1]. Although the situation may have slightly improved from a few years back, we still are not producing enough engineers for the needs of the industry [3]. This gap between the graduating engineers and the required expertise has been predicted to be increasing due to the fourth industrial revolution we are experiencing [4]. This concept paper is part of the research conducted by a 2018 established Nordic initiative related to, among other topics, STEM education and its attractiveness. This initiative comprises a consortium with all Nordic countries as participants. The partners at the consortium are the Royal Institute of Technology in Stockholm (KTH) in Sweden, Aalborg University in Denmark, Aalto University in Finland, Reykjavik University in Iceland, Stavanger University in Norway, as well as the Association of Nordic Engineers (ANE), and NORDTEK (a network of the Rectors and Deans of the Technical Universities in the Nordic and Baltic countries). Focusing on the Nordic countries and their STEM education provides a novel view on this issue while the general aim is to construct an online knowledge hub with the latest information regarding STEM education, teaching, and other related issues. Despite the Nordic view, the aim is to gather STEM education knowledge and conduct studies the results of which may be of global use. Since the decreasing interest in STEM subjects touches all Nordic countries (actually most Western countries), is was deemed essential to form a consortium and use the power of a team to investigate how STEM could be made more attractive and how that information would be disseminated most effectively [5]. Despite many efforts in changing this declining trend of STEM studies, according to Vækstråd [6], these efforts are not enough in order to meet the requirement of qualified staff in 2025, not to mention beyond that. For all Nordic countries, except Finland, TIMMS [7] and PISA [8] results have caused serious concerns regarding young people's performances in science and technology subjects. Despite the fairly good results for Finland [7, 8], they also have declined from the past results. According to some studies (e.g. [9, 10, 11]), poor performance often correlates to low interest for a field, and the fact that our students' performance is decreasing is a cause for concern. In addition to this, a recent study on how higher gender equality, such as present in the Nordic countries, correlates with fewer women choosing STEM fields [12]. Naturally, we do not aim at inequality, but need to find other ways to increase STEM interest in adolescents and females. One of the goals of this Nordic initiative is to present concrete strategies on how to increase the performance and interest in STEM as well as how to enhance the interplay between upper secondary level and university level in order to increase the quantity of young people seeking a career in science and technology. Although all Nordic countries have several outreach initiatives and programs to enhance the attractiveness of STEM education, see, e.g. [13]-[15], it is still not clear what factors are crucial to achieve positive, measurable, long term effects of these types of activities. The aim of this subproject is to focus on the importance of wellorganized and inspiring STEM outreach activities at upper secondary level and how the Nordic countries have succeeded in correlating STEM at the upper secondary level with engineering education and vice versa. Since STEM subjects and their studying is essential for our future within Industry 4.0, the attractiveness of STEM has been explored already for decades. These studies have looked at, for example, outreach activities and their influence [16], gender bias among STEM [17], parents' level of education [18], pre-university engineering education [19], the transition from secondary education to tertiary education [20], and finally even peer influence on academic involvement [21], to name just some of the approaches to this important topic. Building on previous studies on STEM outreach activities, this study aims to identify which elements contribute in the effectiveness of STEM outreach activities, i.e. where's the "beef" in these activities. 2 PREVIOUS STUDIES ON STEM OUTREACH Although the outreach initiatives have been studied to great lengths, the exact impact of any such program defies measuring. We can use pre- and post-tests to collect varying data and we can obtain information on participants attending the offered activities. The influence of outreach activities is only one piece in an adolescent's life where teachers, peers, and home also impact on the path adolescents choose after secondary level [22]. Qualitative studies on outreach activities would require quite notable efforts and the survey questions/interviews would have to be thoroughly planned in order to obtain the appropriate information on these activities and the views of the participants on them. Being attracted to STEM studies requires interest, which is a spark to (internal) motivation and both motivation and interest are necessary to accomplish just about anything [23]. Therefore, finding ways to increase interest in STEM studies should in turn motivate students to pursue studies and careers in STEM fields. The present paper is an initial study on STEM outreach initiatives and it aims to identify elements in outreach initiatives that can be seen as those increasing the attractiveness of STEM. 3 METHODOLOGY To identify the elements that constitute the effectiveness of STEM outreach activities, a literature review of 37 articles was conducted. The studies included in the analysis had to include a clear evaluation of the impact of the activities, i.e., a descriptive study was not enough. Furthermore, studies solely focusing on theory development or different measuring approaches were not included either. Although valuable and informative, also conference proceedings were excluded from the analysis. The excluded 21 studies are provided in Appendix 1. A total of 16 empirical studies with a focus on the impact of STEM outreach activities targeted to K-12 were selected for the analysis. In these studies, the overall objective of increasing students' interest to study engineering was operationalized into one or more elements of effectiveness. The studies were published between 2001–2018 in the following scientific journals: Journal of Engineering Education (5 articles), Global Journal of Engineering Education (2 articles), European Journal of Engineering Education, Studies in Higher Education, Journal of Higher Education Outreach and Engagement, International Journal for Service Learning in Engineering, Humanitarian Engineering and Social Entrepreneurship, CBE-Life Sciences Education, Journal of Science Education and Technology, International Journal of Science and Mathematics Education, and IEEE Transactions on Education. The included 16 studies are provided in Appendix 2. Inductive content analysis (see e.g. [24]) was applied to summarise the informational content of the descriptions related to evaluating the impact of STEM outreach activities on students. The descriptions were categorised according to thematic similarity. The analysis was limited to the empirical part of each study, thus excluding discussion and conclusions from the analysis. Elements that were related to some other target groups than the participating students, such as the facilitators or teachers involved, were excluded from the analysis. The analysis resulted in altogether 17 elements that have been applied in previous empirical studies to evaluate the effectiveness of STEM outreach activities. 4 THE IMPACT ELEMENTS Out of the total of 17 elements that constitute the impact of STEM outreach activities in previous empirical studies, nine were related to the outcomes and eight were related to the design of the activity. Outcomes referred to the intended and achieved results of the activity, such as learning. Design referred to how the activity was organized, including the teachers, facilities, activities, and equipment. Out of the total of 16 empirical studies included in the analysis, 14 focused on evaluating the outcomes and six focused on evaluating the design. 4.1 Evaluating effectiveness from the perspective of outcomes In most of these studies, developing understanding of STEM subjects was the main and often the only indicator of effectiveness. Learning of STEM resulting from participation in the outreach activity was typically seen as a way of encouraging the students to continue studying these topics. Developing understanding is also relatively easy to measure in relation to the intended learning outcomes. It may also be evaluated by measuring students' performance in STEM during or after the outreach activity. Enhanced understanding of and performance in STEM does not automatically improve students' understanding of their own capability to perform STEM-related tasks. Consequently, evaluation of understanding and performance was often accompanied with of measures of STEM self-efficacy. In addition to self-efficacy, evaluating the effectiveness of outreach activities was focused on improving students' attitude towards STEM, which is often seen as connected with motivation to study engineering. Understanding the relevance of STEM was applied as a criterion of effectiveness in reference to both personal relevance and working life relevance. The relevance stemmed from understanding the possibilities of applying engineering to, for example, develop meaningful career paths and tackle personally meaningful challenges. It was also seen to require knowledge of engineering work, including career opportunities and different fields of engineering. While students' knowledge of engineering work may be rather easy to measure, evaluating their enrollment in engineering studies after participating in STEM outreach activities often required a longitudinal approach. Further, tracking down the participants' subsequent engagement in STEM, such as their participation in additional extracurricular science programs, was deemed challenging. Thus, some studies focused on measuring the participants' aspirations, such as their aspiration in pursuing a career in engineering, instead of their actual study choices. To avoid regression of these aspirations over time, the participants should be exposed to several STEM outreach activities during their studies. 4.2 Evaluating effectiveness from the perspective of design In the design of the activity, some of the studies were focusing on innovative learning experiences. These experiences involved student activating hands on learning, exposure to advanced laboratory techniques, participation in hands-on laboratory investigations, clear instructions, informal demonstrations, and overall 'good teaching'. In addition to high quality teaching, the studies emphasized the importance of interesting and authentic tasks. In deciding which topics would be interesting for the students to work with, a real-world connection and personal relevance were highlighted. While enjoyment and positive atmosphere were explicitly evaluated in two studies only, experiencing innovative learning with authentic tasks was often implicitly aimed at making learning of STEM enjoyable. When evaluating the effectiveness of STEM outreach activities from the perspective of their design, innovative learning experiences and authentic tasks were often accompanied with open problem-solving. Working autonomously with complex challenges exposed the participant to dealing with uncertainty and applying an inquiry approach to learning. On the other hand, some studies emphasized the role of teacher support. This category included teacher-student interaction, friendliness of teachers and teaching assistants as well as their availability, knowledgeability and credibility as engineers and scientists. While teacher support was deemed important, peer support and learning from other students and tutors during the process were also evaluated. Peer support could have been critical especially for those student groups, such as young female students, who are often marginalized in the context of STEM studies. For these students, the acceptance of peers could be even more important than the support coming from their teachers. In addition to the learning process, some studies evaluated STEM outreach activities on how well they provided information on STEM careers. Receiving information on career prospects in STEM fields was deemed important for increasing the experienced meaningfulness and usefulness of these studies beyond the immediate course context. This category also involved sharing information on the work of engineers and scientists, engineers' role as problem solvers, and the impact of engineers in the world. Sharing this information could contribute in fighting stereotypes and misconceptions related to these fields. This category involved diminishing the perceived psychological and social costs of engaging with STEM, such as being stigmatized as a 'nerd' or a 'geek', and promoting alignment of STEM activities with participants' identity. The resulting categorization of elements that constitute the effectiveness of STEM outreach activities is presented in Table 1 below. Table 1. Elements that constitute the effectiveness of STEM outreach activities As expected, some of the elements are present in the same outreach activities (e.g. [26, 28]) while others are more focused (e.g. [36, 37, and 41]). 5 DISCUSSION Majority of the outreach activity studies tend to focus on the outcomes rather than on the design of the activities, which has its benefits as the activities often provide information on STEM careers and opportunities while measuring interest in them. Nevertheless, focus on the design aspect would provide teachers and facilitators ideas on efficient ways of organizing various events as well as on what has/has not been tried previously. The results gathered from the literature included in this study, form an image of multifaceted impact on STEM attractiveness. However, influencing on only one aspect of adolescent lives may not have a desired impact, i.e. more STEM students. At the moment, those students who apply to study STEM subjects, would probably do so despite any outreach activities, parental involvement, or their peers. However, much potential is left untapped if those students whose motivation or interest could be ignited are ignored. Similar to understanding and learning, the impact of outreach activities is a challenge to measure. It was evident from the studied publications, which were not able to definitively state which elements brought success in increasing STEM attractiveness – at least in terms of adolescents applying to study STEM subjects. Despite the scrutiny of outreach activities, it proves difficult to distinguish those specific activities that genuinely increase student interest for a longer period of time. Despite the challenging task, the main elements found throughout the articles on STEM outreach activities include authentic, hands-on, real world connection, involvement, and autonomous problem solving in the design end of the activities while in the outcome end interest, motivation, attitude, and understanding are most pronounced. This literature review outlines a framework of outreach activities and their evaluations and, hence, provides the elements necessary for a successful outreach initiative. As is evident from Table 1, many of the studies focus on similar outcomes and elements. Despite this, differences between the activities and their aims are evident. These categorizations indicate how, although looking at similar programmes and activities, there still are notable differences in them. This may be one of the reasons why we still do not have the exact answer to what works in an outreach activity and how we could guarantee positive results. When studying human behavior, their aspirations and development, so many elements are involved that pinpointing to one particular one as the influencer would be quite foolish. Defining the elements influencing STEM attractiveness is not an easy task. Nevertheless, it is an important one and, as the number of students applying to study STEM subjects at tertiary level continues to decline in the Western countries, we need to find the "beef" on how to attract more (female) students to study STEM subjects. Most outreach activity studies focus on the outcomes of the outreach initiative and the most typical study includes pre and post-tests of some sort, naturally collected before and after the organized event or activities. Impact studies are rare and it is also rare to find an article in any of the higher ranking journals. Potentially these two facts are related and if so, the situation begs for a change. Our societies rely on adequately educated people with further technological advances waiting to appear when we least expect it. We do not want the industry to be handicapped due to missing qualified personnel. If the future engineers are to save the world, the engineering educators are responsible for attracting enough students in order to educate them to do so. REFERENCES C. Olofsson, "Positive trend stalled – women still underrepresented in STEM subjects", Nordic Information on Gender [On-line] Available: https://www.nikk.no/en/news/positive-trend-stalled-women-stillunderrepresented-in-stem-subjects/ April 13, 2016 [Accessed January 3, 2019] EducatingEngineers.Com, A site with possible careers for engineers [On-line] Available: http://educatingengineers.com/career-specialties [Accessed January 10, 2019] GetEducated.Com, A site with information on STEM careers [On-line] Available: https://www.geteducated.com/careers/stem-majors [Accessed January 10, 2019] Colebrook, C., Cory, G., Dolphin, T., Doyle, P.,Fox Carney, D., Hatfield, I., McNeil, C., Pontin, G. and Stirling, A. (2015), European jobs and skills A comprehensive review 2015 The JPMorgan Chase New Skills at Work programme,November 2015, [On-line] Available: https://www.ippr.org/files/publications/pdf/european-jobs-and-skillscomprehensive-review-2015_Nov2015.pdf [Accessed January 3, 2019]. Schreiner, C. and Sjøberg, S. (2010), The ROSE project: An overview and key findings University of Oslo, March 2010, [On-line] Available: https://roseproject.no/network/countries/norway/eng/nor-Sjoberg-Schreineroverview-2010.pdf [Accessed January 8, 2019] A report on Danish workforce qualifications in 2016, Rapport om kvalificeret arbejdskraft, Danmarks Vækstråd, December 2016. http://timss2015.org/ http://www.oecd.org/pisa/. Heaverlo, C. (2011), STEM Development: A Study of 6th-12th Grade Girls' Interest and Confidence in Mathematics and Science. Doctoral Dissertation, Iowa State University, available on-line: https://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=1035&context=etd [Accessed January 14, 2019]. Barzanji, T. (2013) Growing the STEM: Encouraging Interest in STEM subjects among low socio-economic Australian secondary students. Harvard Kenedy School Mossavar-Rahmani Center for Business and Governmanet, M-RCBG Associate Working Paper Series | No. 18, available on-line https://www.hks.harvard.edu/centers/mrcbg/publications/awp/awp18 [Accessed January 14, 2019] Smith, C.J. (2016) Math anxiety and low self-efficacy: The effects of math anxiety and low self-efficacy on students' attitudes and interest in STEM. University of Southern California Doctoral dissertation. Khazan, O. (2018) "The More Gender Equality, the Fewer Women in STEM", The Atlantic, Feb 18, 2018 [On-line] Available: https://www.theatlantic.com/science/archive/2018/02/the-more-genderequality-the-fewer-women-in-stem/553592/ [Accessed January 13, 2019]. Information on collaboration with high-schools in Denmark [On-line] Available: https://www.aau.dk/samarbejde/tilbud-til-gymnasier/ [Accessed January 15, 2019]. Information on collaboration with primary education in Denmark [On-line] Available: https://www.aau.dk/samarbejde/folkeskolen/ [Accesssed January 15, 2019]. Divitini M., and Vujosevic, V. ExcITEd Core Project Informed Decision Information on Norwegian pre-university activities on IT fields [On-line] Available: https://www.ntnu.edu/excited/informed-decision [Accessed January 30, 2019] Gumaelius, L., Almqvist, M., Árnadóttir, A., Axelsson, A., Conejero, J.A., García-Sabater, J. P., Klitgaard, L., Kozma, C., Maheut, J. Marin-Garcia, J., Mickos, H., Nilsson, P.-O., Norén, A., Pinho-Lopes, M., Prenzel, M., Ray, J., Roxå , T. and Voss, M. (2016), "Outreach initiatives operated by universities for increasing interest in science and technology," European Journal of Engineering Education, 41:6, pp. 589-622. [On-line] Available: https://doi.org/10.1080/03043797.2015.1121468 [Accessed January 18, 2019]. Farrell L. and McHugh, L. (2017), "Examining gender-STEM bias among STEM and non-STEM students using the Implicit Relational Assessment Procedure (IRAP)," Journal of Contextual Behavioral Science, 6(1), pp. 80-90. [On-line] Available: http://dx.doi.org/10.1016/j.jcbs.2017.02.001 [Accessed January 19, 2019]. Harding, J. F., Morris, P. A. and Hughes, D. (2015), "The Relationship Between Maternal Education and Children's Academic Outcomes: A Theoretical Framework," Journal of Marriage and Family 77. pp. 60-76. Kolmos, A., Holgaard, J.E., Clausen, N.R. and Bylov, S.M. (2017), "Transition from high schools to engineering education," SEFI 2017 Proceedings pp: 998- 1005 [On-line] Available: https://www.sefi.be/wp- content/uploads/SEFI_2017_PROCEEDINGS.pdf [Accessed December 20, 2018]. de Vries, M.J., Gumaelius, L. and Skogh, I.B. (Eds.) (2016), Pre-university Engineering Education, Sense Publishers. Wanga, M.-T., N. Kiuru, N., Degol, J. L. and Salmela-Aro, K. (2018), "Friends, academic achievement, and school engagement during adolescence: A social network approach to peer influence and selection effects," Learning and Instruction Volume 58, pp 148-160 [On-line] Available: Vasalampi, K., Kiuru, N., and Salmela-Aro, K. (2018), The role of a supportive interpersonal environment and education-related goal motivation during the transition beyond upper secondary education. Contemporary Educational Psychology Vol. 55 pp. 110-119. Hidi, S. and Renninger, A. (2006), The Four-Phase Model of Interest Development. Educational Psychologist Volume 41 - Issue 2, pp. 111-127. White, M. & Marsh, E. (2006). Content analysis: A flexible methodology. Library trends, 55(1), pp. 22-45. Jensen, F. and Sjaastad, J. (2013), A Norwegian out-of-school mathematics project's influence on secondary students' STEM motivation. International Journal of Science and Mathematics Education. 11 pp 1437-1461. Vennix, J., Den Brok, P., & Taconis, R. (2017). Perceptions of STEM-based outreach learning activities in secondary education. Learning Environments Research, 20(1) pp. 21-46. Vennix, J., den Brok, P. and Taconis, R. (2018), Do outreach activities in secondary STEM education motivate students and improve their attitudes towards STEM?, International Journal of Science Education, 40:11, pp. 12631283. Laursen, S., Liston, C., Thiry, H., & Graf, J. (2007). What good is a scientist in the classroom? Participant outcomes and program design features for a shortduration science outreach intervention in K-12 classrooms. CBE-Life Sciences Education, 6, pp. 49-64. Levine, M., Serio, N., Radaram, B., Chaudhuri, S., and Talbert, W. (2015), Addressing the STEM Gender Gap by Designing and Implementing an Educational Outreach Chemistry Camp for Middle School Girls. Journal of Chemical Education, 92 (10), pp 1639-1644. McLurkin, J., Rykowski, J., John, M., Kaseman, Q., and Lynch, A.J. (2013), Using Multi-Robot Systems for Engineering Education: Teaching and Outreach With Large Numbers of an Advanced, Low-Cost Robot. IEEE Transactions on Education, Vol. 56, No. 1, pp. 24-33. Innes, T., Johnson, A. M., Bishop, K. L., Harvey, J., & Reisslein, M. (2012). The Arizona Science Lab (ASL): Fieldtrip based STEM outreach with a full engineering design, build, and test cycle. Global Journal of Engineering Education, 14(3), pp. 225-232. Goodyer, J., & Soysa, I. B. (2017). A New Zealand national outreach program– inspiring young girls in humanitarian engineering. International Journal for Service Learning in Engineering, Humanitarian Engineering and Social Entrepreneurship, 12(2), pp. 1-14. Sadler, K., Eilam, E., Bigger, S., & Barry, F. (2018) University-led STEM outreach programs: purposes, impacts, stakeholder needs and institutional support at nine Australian universities, Studies in Higher Education, 43:3, pp. 586-599. Anderson, L. S., & Gilbride, K. A. (2003). Pre-university outreach: Encouraging students to consider engineering careers. Global Journal of Engineering. Education, 7(1), pp. 87-93. Poole, S., de Grazia, J. & Sullivan, J. 2001. Assessing K-12 pre-engineering outreach programs. Journal of Engineering Education, 90(1), pp. 43-51. DeGrazia, J. L., Sullivan, J. F., Carlson, L. E., & Carlson, D. W. (2001). AK12/university partnership: Creating tomorrow's engineers. Journal of Engineering Education, 90(4), pp. 557-563. Markowitz, D. G. (2004). Evaluation of the long-term impact of a university high school summer science program on students' interest and perceived abilities in science. Journal of Science Education and Technology, 13, pp. 395-407. Colston, N., Thomas, J., Ley, M. T., Ivey, T., & Utley, J. (2017). Collaborating for Early‐Age Career Awareness: A Comparison of Three Instructional Formats. Journal of Engineering Education, 106(2), pp. 326-344. Moskal, B. M., & Skokan, C. K. (2011). Supporting the K-12 classroom through university outreach. Journal of Higher Education Outreach and Engagement, 15(1), pp. 53-75. Moskal, B.M., Skokan, C., Kosbar, L., Dean, A., Westland, C., Barker, H., Nguyen, Q.N., & Tafoya, J., (2007). K-12 outreach: identifying the broader impacts of four outreach projects. Journal of Engineering Education, 53(1), pp. 173-189. Smaill, C.R., 2010. The implementation and evaluation of a university-based outreach laboratory program in electrical engineering. IEEE Transactions on Education., 53(1), pp. 12-17. Fantz, T. D., Siller, T. J., & Demiranda, M. A. (2011). Pre‐collegiate factors influencing the self‐efficacy of engineering students. Journal of Engineering Education, 100(3), pp. 604-623.