Evaluating Science Outreach Impact

1 Introduction

What are Science outreach events?

From a public perspective, a science outreach event is an opportunity to bring researchers closer to the general public and increase awareness of their research (1). Work of scientists is often viewed as a mystery to the general public, yet, it provides answers to the most persistent challenges our societies face, including climate change, public health and food security (2). Hence, showcasing the impact of researchers work on people´s daily life not only supports the public recognition of scientists but also helps to increase the public understanding of specific issues, which can lead to more informed choices in our society (3). From a school perspective, a science outreach event aims to inspire young people’s minds and attempts to engage participants with the idea of becoming a future scientist. Universities across the UK offer extensive programmes of talks, workshops and events for primary and secondary school children to raise awareness of science careers and to break stereotypical images. Stereotypes commonly attributed to Physics that claim it is a masculine subject have been ingrained into students from a very early age onwards. Hence, several literature sources argue that science outreach interventions need to be more effective at removing societal obstacles that prevent equal gender distribution in science (4). Yet, up until this current date, little research has been conducted on the effectiveness of outreach interventions to attract students into sciences. Science departments at Universities across the UK involved in outreach events are interested to know if they should be working more closely in partnerships with schools or host more one-off events. It remains unclear if Universities should expand their outreach activities towards students at the primary level, or if it is more effective to address students at the secondary level when they are confronted with their subject choices. Furthermore, no more than a handful of literature sources report which types of interactions or activities at outreach events have an impact on students in their decision to further pursue a science subject in school or at University level. Hence, more effective evaluation of outreach events is critical for improving individual activities of outreach programs, and for reporting back to science departments at Universities or funding partners.

Aim of this report

This report aims to investigate the impact of science outreach events targeted at school students, specific to recruitment to higher education and more broadly. Part of this investigation is to explore the types of assessment strategies that social researchers have been using for the evaluation of science outreach events. It will be defined what science outreach events are and what challenges exist in measuring their impact. Scientific data to explore the effects of outreach events will be obtained from different literature sources and through feedback from three groups: high school Physics teachers, STEM Ambassadors and first-year undergraduate students studying Physics at Glasgow University. To explore reasons for bringing students to outreach events, quantitative and qualitative data will be collected from an initial survey of a pilot group of teachers after bringing pupils to the Quantum Technology School (QTS), which is a two-day senior high school Physics event organised and hosted at the University of Glasgow. Next, a second more focussed teacher survey will be launched to a wider audience to gain a more in-depth understanding of the effectiveness of the types of activities in reaching students and to understand how outreach events help teachers in their job. Furthermore, one-on-one telephone interviews with STEM Ambassadors will be conducted to investigate their viewpoint on outreach events, and more specifically to understand the impact and effectiveness of their outreach towards high school-aged pupils. Results of questionnaires and in-depth focus groups with the first year undergraduate students will investigate students´ perceptions of whether attending an outreach event impacted their choice to study Physics. The interviews seek to analyse if there were causes for potential obstacles to students attending outreach events before coming to University or if there were any correlations between having a role model who is working in a science field known personally to the student and their choice to attend an outreach event. The report will also have a section on recommendations that aims to advise outreach event organisers on the types of activities that have been enjoyed most by students. This includes suggestions of lengths of time of an activity, the types of presenters and the venues. Finally, the report will provide examples of assessment strategies that will help to measure the impact of outreach events effectively. This will serve to contribute to the establishment of a research base for on-going improvement and enrichment for outreach event organisers.

Objectives of Science Outreach

Promoting science

Outreach events provide the opportunity for school students and the general public to experience first-hand what scientists do daily, how they conduct research and what impact this has on society (5). “Doing science outreach events helps to promote the potential, pervasiveness and ubiquity of the science”, according to the Institute of Photonics (6), which is one out of many non-profit research institutes that have created a portfolio of science outreach programs and activities targeted at all areas of society. Universities, governmental, industrial institutions, and informal science institutions such as museums and planetaria have sought to bridge the gap between scientists and the general public as science has become increasingly relevant and important to society at present day. Science is prevalent in all facets of our lives in today’s world”, said Erika Shugart, science event manager at American Society for Cell Biology (ASCB), “it is essential to assist the public to be able to understand the basic scientific concepts to help them make more informed decisions about personal, health, climate or business-related decisions.” In addition to public engagement, conscious efforts have been made by various organizations, private and governmental, to promote science in schools to prepare the next generation of citizens. “Physics Without Frontiers” is an example of such an organization that works in close co-operation with schools to inspire, train and motivate students in Physics and Mathematics worldwide to help build the next generation of scientists (7). The philosophy of the organization is that progress of knowledge, skills and attitudes in science-related fields is essential for economic growth, citizens health, national security and the stability of the nations. In the UK major science festivals including "The Big Bang fair" or "The Science Week celebration" is promoted not only by Universities and large companies such as BAE Systems, Siemens or Jaguar Land Rover but also the UK government. This is due to the growing national concern that not sufficient young scientists currently exist to sustain the society of the future to face challenges associated to climate change, dwindling resources, the evolution of social media and many more. It is argued that inspiring future generations to engage in STEM subjects is crucial and beneficial for a young person’s educational development and the future of the country (7).

Identify and break barriers to participation

Science outreach events seek to identify and break barriers to participation, which have been identified to be due to societal stereotypes that science is more suited for men. Among the different science disciplines, Physics is considered to be especially exposed to stereotypical views in society. Some research studies state that stereotypes of Physics being conceptually difficult and only suitable for bright male pupils are ingrained in students from a young age onwards (31). It was also found that in many developing countries the pursuit of a career as a female Physicist is frowned upon due to traditional social norms (8). In response to these findings, collaborative attempts have been made by several governmental institutions, such as the Physical Science Study Committee in the US, to tackle these pervading gender stereotypes in schools that deter females from choosing sciences as a career field. An approach widely used in the USA to improve the perception of sciences has been through inviting girls to week-long science camps to explore technology-dependent careers in which women are heavily underrepresented such as Quantum Physics, robotics or commercial aviation. A lot of these summer camps specifically motivate and incentives girls from low-income families to come by offering them scholarships.

To raise science literacy and diversity of the science workforce

A further objective of science outreach events is directed towards increasing the participation of under-represented groups including females and low socioeconomic status (SES) students and to establish an inclusive, sustainable, and diverse science community (4). The shortage of female scientists and the increasing demand for a technologically competent workforce have made British companies, policymakers, and educators come up with different strategies to encourage girls to pursue a career in science. In 2016, the Scottish Funding Council (SFC) published their Gender Action Plan (GAP), which outlines its requirements for all Colleges and Universities to consider, and put in place a plan to address, specific disciplines with a greater 75:25 gender imbalance of Scottish Domiciled Undergraduate Entrants (SDUE) (9). At Glasgow University, there is on-going female underrepresentation in the recruitment pipeline in particular for Physics, which has led to the incorporation of a variety of diversity programmes such as the Quantum Technology School (QTS) outreach event intending to reduce the gender gap in Physics. Furthermore, the University of Glasgow has been active in Project Juno, which is a scheme to recognise and reward departments for developing good practice for an equitable working culture (10). As shown in Figure 1, the gender gaps in Physics 1 and Astronomy 1 have been gradually closing. In 2018, there have been 50% of females for the first time in Astronomy. In other institutions, there is not such an increase in females wanting to study Physics.

Although science outreach programs have proven successful, it remains unclear which types events or activities have had most of the influence to make the gender gap close. Are females more influenceable at open days? There is room for more encouragement and improvement, Public Engagement Officer Zara Gladman believes (11).

Challenges of outreach events

According to social researcher K. Sandra, there is an underdevelopment of outreach events and those that are done are not sufficiently publicized (12). This view is largely supported among educational researchers, who claim that the majority of science outreach are not publicly announced, but are only publicized on the intranet of a University or school website for example (13). This restricts access to privileged students, who have been selected only because of their subject choice or the school they go to. Also, the impact of outreach events is not always evaluated depending on the formality of the event, the priorities of organisers, the types of funding partners and the available time and financial resources (14). According to Alan J. Friedman, outreach organisers of informal events in schools with no funding partners are frequent volunteers, who are not interested or do not see sufficient value in recording the impact either qualitatively or quantitively (15). A study by Sally Thompson et al. found that 76% of volunteers are interested in using outreach evaluation, but only 41% have the necessary expertise and resources, as shown in Figure 2 (16).

In her study, she used an online questionnaire that was sent out to several thousand volunteering science activists, who have been advocating for a better food diet and lifestyle all over the world. It was found that many outreach activists were unaware of assessment strategies to evaluate the impact of their activities. Another challenge for outreach programmes is how to go about with preparing teachers to guide their students in research and other different forms of inquiry when the backgrounds of the majority of the teachers in science are limited. The spread of innovations is also affected by the organisational and professional contexts in which teachers work. Schools, however, unfortunately, do not provide the correct opportunities, resources and incentives for teachers in which they could be able to share such practices that would create environments that are fertile for the growth and spread of curriculum innovations. While networking and support for outreach programs could be provided by teacher professional organisations, it is only a subset of the teachers who are more active who participate. Educators’ political context is also very important. While the intention for the development of cooperative extension was to provide information to those individuals who operated in relatively neutral environments, schools operate in contexts that are highly politicised. Makers of policies, education researchers and practitioners have not been able to build a consensus on the overall goals which has gone a long way in creating barriers to expert-driven models of dissemination. For instance, inquiry-based learning is promoted by agencies like the National Science Foundation while at the same time the "No Child Left Behind" act of 2001 makes an attempt to impose testing that is standardised that is not capable of promoting inquiry. As such, in contrast to farmers, who could go out of business when not capable of adopting innovations, teachers could also be penalised whenever they adopt such innovations that are not capable of increasing their student`s test scores.

Outreach events

Types of outreach events

Science outreach events can take many forms designed for many types of audiences by a variety of different organizations. Professional science outreach events are typically hosted by research institutes, universities, or other science institutions such as science museums, which tend to receive support from official funding partners. The funding agencies are specifically interested in making the involvement with outreach programs as efficient as possible through effective use of the existing resources (17). Hence, these formal events tend to have hired professionals, whose responsibility it is to measure and evaluate the impact of the outreach event. A dedicated team of social research scientists design social research methods such as a questionnaire to target the intended audience. An organisation with more financial resources has the advantage to obtain external consultancy, training and information from professionals, which can significantly enhance the effectiveness of public outreach evaluation. Less formal outreach events have recently grown in popularity, especially in major cities. Café Scientifique is an example of such an event that is held in cafes or bars, where everyone is welcome and may listen to the latest scientific ideas or technological inventions for the price of a beverage. Guest scientists give presentations about their work, share scientific ideas or take part in discussions with members of the public. Science cafes typically have several dates a month, where any scientist is allowed to share their ideas. As mentioned in Section 1.4 of this report, evaluation of the impact is often not measured at these informal outreach events due to a lack of time, expertise, financial resources, or several other barriers that the presenting scientist faces. Improving technology and faster internet connections in schools allow for virtual outreach sessions. An initiative by STEM Learning, called “Skype the expert” permits STEM Ambassadors to stay at home or work and video call a classroom in a school via Skype. This program helps especially those schools which are situated in the countryside or further away to communicate with scientists. Although the use of Ipads in the classroom has been viewed critically, from a science engagement point of view these devices can be highly useful and powerful when implemented appropriately into the school curriculum (18). Using their Ipads, students can easily access YouTube science channels, view posts or read blogs of scientists to be inspired.

Types of Activities at outreach events

Lecture-type presentations

To effectively reach a high number of participants at outreach lectures, scientists need to carefully consider the scientific background of the audience and plan the level of scientific concept and word choice accordingly (19). Differences in age, culture, language, or mental or physical disability can impact how someone engages with science and need to be taken into consideration. A recent study analysed open-public lectures given by Brian Cox, who is a renowned Physics Professor and internationally known as a presenter of his BBC science programmes. It was found that he managed to embrace the heterogeneity of his audience through using easy, everyday language and incorporating stories to hook students´ imagination (20). This finding is in line with numerous UK national surveys reporting that speakers need to embed their theory in stories or context for students to construct meaning out of the abstract and create a bridge to their personal lives (21). Another advise from Physics guest lecturer, Aoife Monks, is to avoid overloading presentations with too much information especially for younger students, whose attention span tends to be shorter (22).

Panel of scientists

Panels that promote discussion at science outreach events are useful to trigger an exchange of viewpoints from different scientists (23). These involve on-the-spot interactions and are hence more difficult to prepare for than presentations. A moderator is needed to introduce the speakers to ensure equal speaking time and direct the team. Lily Strelich, who was involved in organising a science outreach event, connecting 28,000 participants with a panel of geoscientists, argued that a panel of scientists answering questions can be useful to show the diversity of scientists (24). The audience can see a variety of STEM role models rather than only one speaker, which influences the audience to an open-minded attitude and respect for other's opinion (25).

Interactive workshops

“Tell me and I’ll forget; show me and I may remember; involve me and I’ll understand”, is a Chinese proverb, which captures the key value of interactive workshops in the context of outreach events (26). Several studies have shown that interactive workshops are beneficial for students to get engaged and involved in the activity and for establishing a community atmosphere at the event. Presenters can step back and take the role as facilitators.

Role of the Presenter

The novelty of a presenter being different from the regular teacher can be a benefit. A recent study showed that students´ showed better behaviour and paid more attention to someone different from their everyday teacher (4). However, a recent study conducted by the IOP about preferences of presenters at outreach events showed opposite results (27). The individual personality and gender of the presenter were rated to be unimportant, more essential were the qualities of the presenter. Another study by the “Alliance for Excellent Education” showed that the participants mostly valued the presenter´s ability to establish a positive learning environment, in which students could feel confident about expressing their views (28). Yet up until now, it has been largely unexplored in the literature, what impact it had on the perception of audience when the presenter or the demonstrator was a female. The female audience might not identify themselves with the gender, but elude something completely different such as a person, who works all the time. A question to investigate further in this report is, if students are more engaged, because the presenter is different from the regular teacher or because the presenter is a specialist.

Methods to measure the impact of outreach events

Evaluation methods

The literature proposes several formal evaluation methodologies such as the "Logic model" and the "Theory of change" that have been developed in the late 20th century with a rigorous epistemological foundation and honed over multiple studies. Other models such as the "Kirkpatrick model", the "Realist evaluation" or the "Evaluative case study" have also gained their popularity in evaluating the social impact of an outreach event.

The following section will describe the different theories to measure the impact of outreach events.

Types of models

Logic model

To measure the impact of outreach events, the National Co-ordinating Centre for Public Engagement (NCCPE) proposed the use of logic models that researchers at outreach events can use to map out the approach of their activity (29). These involve graphically illustrating the causal pathways of the inputs and activities as illustrated in the following diagram:

Clear predictions are made before the event about the impact of each of the planned activities. Researchers need to develop the logic model together as a team; each researcher has to specify the benefits participants will take away from their activity. This can be a cascade of effects, for example, a concrete skill that participants will learn, an increase in knowledge about a particular area of science, a gain in confidence to use technology or a change in the general attitude towards scientists. "The intention to create a logic model as a tool should be an integral part", argues Dr Lucy Davies, Quality Officer from the University of Plymouth, "as it optimises the communication between each of the presenters of the event." Short, medium and long-term outcomes are formulated from the start on, which makes it easier at the end to come back for a detailed evaluation.

Theory of Change

The Theory of Change first emerged in the 1990s to analyse and evaluate complex programs and initiatives working for social and political change (30).the theory is essentially a comprehensive illustration and description of how and why desired changes are expected to come about in particular contexts. The particular focus of the theory is on mapping out and filling out the missing middle between what is done by a change initiative and a programme and how these lead to the achievement of those goals that are desired. This is achieved through, first, the identification of the desired long-term goals which is followed by working back from that for the identification of all those outcomes and conditions that need to be in place for the achievement of the goals. An outcomes framework is used to map out all this. the Outcomes Framework then provides the basis for the identification of those types of interventions and activities that would lead to the achievement of those outcomes that are identified as preconditions for the achievement of long-term goals. With this approach, it becomes easier to understand the precise link between activities and the achievement of long-term goals. With such, an opportunity is created for better planning, in that activities are linked to an understanding of how change comes about that is properly detailed. In contrast to logic models, the Theory of Change begins with its long-term goal and then works backwards rather than engaging in forward-oriented "so-that" reasoning. The Theory of Change is interested in how well a precondition needs to be met to achieve the next goal. Figure 4 provides an example of a Theory of Change Model, showing the outcomes, activities and resources of an outreach event aimed at closing the gender gap in Physics in the UK:

For the reader to comprehend the following explanation, the units of the logic strand described are coloured in red in Figure 4.

First, the presenting scientists would have to find scientific evidence suggesting that there is a gender gap in Physics in the UK. For example:

“The IOP found out that Physics has one of the highest gender gaps in science according to an analysis of more than 36 million authors of academic papers over the last two decades.” [16]

Next, the researcher would need to back up how the medium-term outcome i.e. “raising the confidence” of females would lead to the long-term outcome. For example:

"Gender disparities in performance do not stem from innate differences in aptitude. Instead, a study conducted by the Organization for Economic Cooperation and Development (OECD) found that girls are particularly vulnerable to low self-confidence and are more likely to have a low attitude towards Physics. By improving their attitude and confidence, girls would be more likely to consider a career in Physics.” [17]

Then, the presenting researcher would have to find evidence that the short-term outcome i.e. promoting communication skills of girls is likely to result in the medium-term outcome. For example:

“A study by Chowdhury et al. showed that effective communication skills can raise confidence and self-efficacy.” [18]

Next, the presenting researcher would have to find evidence that teamwork and creative stories promote communication skills.

“A study conducted by the IOP found that girls like to work in teams…. They enjoy being creative and like writing stories.” [19]

The presenting scientists would map out in detail the resources and activities that they would use to help achieve the immediate outcomes and back up their assumptions with supporting scientific evidence. For example:

“Science newspapers can provide students with new story ideas, which can help them to come up with ideas for creative writing.” [20]

Kirkpatrick model

The Kirkpatrick model, originally designed to evaluate professional training courses for employees in enterprises in the United States, has also gained acceptance for measuring the impact of outreach events (31). The model is based around four levels of evaluation, as shown in Figure 5.

The first level Reaction: measures the ways through which participants react to training offered to them, for example, satisfaction. The reaction can be assessed through informal feedback, measuring how participants reacted to the activity, which is an evaluation carried out immediately after the event. Participants are asked to which degree they found the activity favourable, engaging and relevant to their life situation. Level 2: Learning: analyses whether the participants grasped and understood the training. This tests the degree to which participants acquired the intended knowledge, skill or attitude. This stage of evaluation requires more time and effort as a comparison is needed to assess the knowledge situation before and after the activity.

Level 3: Behavioural Change looks at how participants can apply the learnt knowledge at their job, in school etc.

Level 4: Result evaluates if the event resulted in behavioural changes that impacted the person´s life.

Activity planning should begin with Level 3 and 4, which is often neglected in outreach evaluation practices. As outlined by the Kirkpatrick model, evaluation needs to begin at the first level, after which as allowed by available resources and time, should continue orderly through the other three levels. It is possible to use the data from all previous levels as foundations for future analysis. As a result of this, every other subsequent level provides a measurement that is even more accurate of the training`s usefulness and also calls for a more demanding and time-consuming evaluation simultaneously. While the Kirkpatrick model still remains popular, its adoption should be done with care. As of today, the model is over sixty years old and there have been very many changes over the years in the way things are done. There are other forms of training today that would be more effective. It is worth noting that today, with the rise of user-directed learning that is personalised to the individual needs of different learners, the prominence of formal training has been declining. Kirkpatrick`s model is not suited to the new learning approach necessarily. Another drawback of the Kirkpatrick`s model is that the behavioural change and result levels which with no doubt yield the most important information, consume a lot of time in addition to requiring a lot of resources and their implementation is also quite expensive. This makes the model not suitable for all situations, for example, one-off training.

Social Research

Qualitative and quantitative social research methods

Qualitative and quantitative social research methods are used as a means of providing evidence to examine ideas about society and to test hypothetical theories about groups of people (32). Quantitative observation methods are sources of data collection to quantify attributes, attitudes, behaviours or other defined variables with a motive to either support or reject a hypothesis (33). To obtain quantitative data, researchers can employ different research instruments such as questionnaires or polls or systematic observation methods such as counting of the number of people present attending an event. Qualitative observation methods are used to gather non-numerical data such as underlying ideas, motivations or feelings. The most common data collection methods in qualitative research are interviews and focus groups, which provide the potential for more richness of responses. Compared with online questionnaires, focus groups and interviews are more time-consuming. Moderators can greatly impact the outcome of a focus group discussion by injecting their personal biases into the participants´ exchange of ideas (34).

Limitations of social research evaluations

During social research, different factors affect the reliability of the studies, which prevent measurements to be exactly repeatable or replicable. Diverse studies report that participants tend to modify aspects of their behaviour in response to their awareness of being observed, which is known as the Hawthorne effect (35) (36). Research by Mayo E. et al. investigating methods of increasing productivity of workers in an electrical company called Hawthorne Works in Chicago in the 1920s concluded that employees’ productivity increased through the psychological stimulus of being singled out or when the feeling occurred of being given increased attention (37). Participants at outreach activities often receive clear signals that they are expected to have a profound interest in a particular subject or appreciate extended education. Regardless of the content of the activity, the responses given at surveys or interviews are likely to reflect these expectations to a certain extent (38). The Priming effect is another psychological phenomenon, which in the context of the evaluation of outreach events refers to the tendency of participants to rate an activity more positively the nearer in time they are asked about it, especially when the most enjoyable parts of the activity came last. Thus, a survey completed immediately after an outreach event has a higher probability of being rated more positively particularly when being asked for example if they are planning to come back the subsequent year. The Dunning-Kruger effect is a cognitive bias that tends to affect the survey results of those people who overestimate their level of knowledge before the activity because they did not know what to expect. Kruger et al. showed that after particularly intellectually challenging activities participants, who did not have any reference or mean of comparison before the activity, stated in subsequent surveys that they knew less than they had thought and felt that their confidence to succeed had fallen after the event (39). This social phenomenon might occur to for example younger male students at science outreach events, who are feeling confident at the beginning, but as they are repeatedly faced with too challenging tasks, they become more frustrated and are unlikely to desire to re-engage in the given activity. Self-selection bias arises as certain categories of people are more likely to engage with questionnaires than others. Stephen R. Porter et al investigated the correlation between response rates of surveys with the individual´s demographic background, academic achievement and personality profile in the US (40). In his study, he found that the survey respondent was more likely to be female, socially engaged, educated and affluent person. A study by Amany et al. confirmed that female and educated individuals tend to engage more fully with questionnaires. Her study additionally reports that male participants were more likely to respond to surveys if they received a reminder, and older participants were more likely to respond if they were promised a reward (41). With outreach activities, it has been found that those participants who benefitted more from an activity, are more likely to take the time to share their data and give feedback. Thus, researchers have to treat survey results with caution as they can become skewed, if certain categories of people do not take them seriously or do not engage with them at all. This is a concern for social research scientist as it can compromise the validity of the results.

Recommendations to avoid bias in survey instruments

“Although scientists often have no deliberate intentions to create any form of bias in their research, it can easily happen through poorly designed surveys”, states Ana-Maria Šimundić, President of the Croatian Society of Medical Biochemistry and Laboratory Medicine. To avoid bias in survey instruments and to make fair evaluations, social researchers are advised to think carefully about the layout of questionnaires, the wording, and the order of the questions. A study by Bernard C.K. Choi et al. systematically reviewed the design of questionnaires in the public health research domain (42). His study found that bias frequently arose from poorly designed questions, the number of questions asked, the range and type of response categories and from the instructions given to the respondents. The most frequently committed mistake in questionnaires leading to bias was caused through poor question phrasing by the survey designer. This could be caused by ambiguity in the wording such as: "Is your work made more difficult because you are expecting a baby?”. A "no" answer may mean, "No, I'm not expecting a baby," or "No, my work is not made more difficult". Hence, according to Bernard C.K. Choi, good practice to avoid bias in surveys is to ask neutrally worded questions that abandon technical jargon and if needed include footnotes that define or clarify the meaning of the question in its context. Another advice for multiple-choice questions is to go with balanced questions that have an equal number of positive and negative options from which the respondents can select their answer. Following example shows an unbalanced question, because there are three positives and two negatives answers:

How did you like the activity?

Excellent| Very good | Good |Poor |Very Poor

To make the above question balanced:

Very good| Good |Average |Poor |Very Poor

Furthermore, when respondents are asked to select a frequency, questionnaires must provide a scale that does not have any overlapping or missing intervals, which could bring about confusion. Regarding rating scales, Gail M. Sullivan, Professor of Medical Education at University of Connecticut School of Medicine, recommends the use of Likert scales, which is the most universal method for survey collection (43). It consists of a 5- or 7-point ordinal scale. In each question, a statement is presented in which a respondent is asked to indicate a degree of agreement or disagreement in a multiple-choice type format. The responses about attitudes, beliefs and opinions are not a concrete yes or no, which would force the participant to take a stand on a particular topic but allows them to respond in a degree of agreement. Neutral or undecided feelings of participants are equally valid.

Methodology

Initial teacher survey

An initial teacher survey, attached in the Appendix, was issued at the Quantum Technology School (QTS) event on the 11th and 12th September 2019 targeted exclusively at the QTS teachers to understand their motivations for bringing students to the event. The event itself was organised by the University of Glasgow for Higher and Advanced Higher students from schools located around Glasgow. Hard copies of questionnaires, accompanied by a plain language statement, were personally handed out to a total of 20 teachers, who were asked to answer seven questions. The first four questions were asking about the value of outreach events for students and teachers, and about the types of activities that teachers thought were most useful for the students to continue with Physics. The fifth question explored teachers´ decision criteria to choose students in their class to bring to the event. The last two questions aimed at identifying the hurdles and factors that prevented individual students and schools from attending outreach events.

Follow-up of the teacher survey

An additional teacher survey was launched containing 10 questions to follow-up with the information teachers had provided in the previous survey and to gain a more in-depth view of the impact outreach activities have on school students and their decision making towards studying STEM subjects. It was decided to broaden the group of respondents to obtain more responses and thus improve the statistical analysis. The questions were about the attitudes and motivation, in-school and out-of-school events, and the impact of female demonstrators. The difficulty of teachers to respond to surveys arose when students had reported different opinions to them.

Undergraduate survey

First-year undergraduate students enrolled in the Physics 1 (PHYS1001) course at the beginning of the academic year 2019-20 were asked to fill in a survey, which is attached in the Appendix. It was estimated that the class will be between 250 and 300. The surveys were handed out as paper copies by Dr Sneddon, or an appointed alternate, in first-class meetings (September 2019) alongside a plain language statement. The questions explored participants’ motivations for studying Physics at the University of Glasgow, and if they had been engaging with outreach events before University. The survey gathered data on what experiences students had at school that led to them opting to take physics at University. All participants were drawn from the membership of the course Physics 1.

Focus Groups with Physics first-year undergraduate students

Three different groups of 17 first-year undergraduate students were asked further questions regarding their attendance of outreach events and the impact these may have had on them to pursue Physics at University. The first two groups consisted of three males and three females and the last group of three males and two females. The students belonged to the same group of Physics first-year students, who had filled in the undergraduate survey a few weeks prior. Because students were asked during their lab sessions if they wanted to contribute to this study voluntarily, a few students refused because they did not want to fall behind with their work.

Interviews with STEM Ambassadors

Interviews were conducted with a group of STEM ambassadors to discuss more open-ended questions. The group consisted of six Physicists from Scotland, who were all from different age and level of experience levels and hierarchical ranks in Physics. A structured interview guide was used to ensure similar information was collected in each interview, which allowed for different questions to be answered by the individual scientists in more detail. Discussed were reasons for scientists to become STEM Ambassadors, what they take away as a presenter, the gains for teachers and students to attend outreach events, how STEM Ambassadors identify changes in attitude of students after an outreach event, at what ages they think students are most likely to benefit from a science program and if they evaluate their outreach activities. Full transcriptions of all STEM ambassador interviews were made using audio recording devices. Any phrases that were repeated by more than one STEM ambassador and that had appeared in the literature were highlighted and labelled as relevant. General categories were derived from the evaluation aims such as “student motivation”. More specific categories were created from multiple readings of the raw data such as “motivation to pursue a Physics career” or “motivation to study at Physics at University” or “motivation to attend another Physics outreach event”. During qualitative coding, one segment of the text might be coded into more than one category. For example, when asked about the value of outreach events to students, a STEM Ambassadors answer was: “students gain insight into University, spend time on campus and interact with peers from other schools”. The answer touched on “student´s personal development”, “University choice”, “extra-curricular teaching” and “encouraging friendships” and “improving teamwork skills”. The transcriptions were subsequently analysed by systematically coding the obtained data. To raise awareness of the study and attract volunteers, publicity was done through different social media platforms including Twitter, Facebook, and Yammer using a private account and the Glasgow University School of Physics and Astrophysics account.

Evaluation and interpretation of results

Comparison of the teacher survey findings with contemporary literature sources

Learning outside the classroom

The teacher survey found that outreach events are valuable and beneficial to students as they learn science outside the classroom in an informal learning environment. Two teachers said: “outreach events give students a valuable learning experience outside the classroom”. Other individual responses were “events like this raise awareness of the world outside school”, “Students´ interest, engagement and determination to continue in Physics increases by orders of magnitude when they experience Physics outside the classroom" and “outreach activities encourage learning in a more informal context outside school”. Feedback obtained from teachers largely coincides with the analysed literature sources, especially for outreach events that adopt a more informal, but yet structured learning approach (44). Several studies exploring the benefits of outreach events concluded that students are more likely to experience a less-controlled environment in "out‐of‐school" settings. Being away from the typical classroom environment surrounded by " real scientists" gives students the feeling of more autonomy and the impression that they are allowed to make their own choices about how to solve a problem (45). Feedback from one teacher was: “students can deepen their understanding and obtain an overarching view of Physics". Marc Behrendt, Professor at Ohio State University, is critical that the majority of students in high schools are ready to benefit from such self-directed independent learning environments at outreach events without prior pedagogical preparation or training of the students (46). For students to adopt a proactive mindset, they need to know what to expect at the event, the purpose of the trip and what consequences they will face if they misbehave. A lack of preparation before arrival can make students feel disoriented at the venue, quickly leading to disruptive unrestrained behaviour. However, a study by Ofsted showed that when teachers planned and prepared students well, learning outside the classroom contributed significantly to raising standards and improving pupils' personal, social and emotional development (47). This result was presented in a report after evaluating the impact of learning outside the classroom of 12 primary schools and 10 secondary schools. Linda Darling-Hammond, Professor at Education Emeritus at Stanford University, agrees and comments: “once teachers have completed their duty of preparing students, it is largely in the hands of the presenter of the outreach event to guide and engage the students” (48). As long as students receive support in meeting their basic needs to study independently by the activity instructor, they will indeed greatly benefit from outreach events (49).

Extra-curricular knowledge

A further finding of the teacher survey was that students gain extra-curricular knowledge and interact with “real-world” scientists at outreach events, which is beneficial to students as they are being taken outside their “bubble”. Feedback from teachers was: “Students gain insight into the leading-edge technology and interact with peers from other schools”, and “students meet specialists in the fields, which helps students gain networking skills”. This finding is in line with some views and social research studies reviewed in the literature. For example, Wayne Jarvis, a consultant at STEM Learning suggests that taking students outside the classroom can lead to a deeper understanding of challenging concepts beyond the national curriculum and provide students with a context for learning in many areas (50). Education Scotland agrees that outreach events can make the curriculum more coherent and meaningful from the learner´s point of view while preparing students with real-life examples for life after school (51).

Career in science

Although students may learn new vocabulary, the results from the teacher survey suggest that the larger impacts of outreach events are on students´ interests and enjoyment of science, which can positively influence students´ decisions to want to pursue a career in science. One teacher comment was: “Outreach events open students’ eyes to possible career options, which provides context and motivation to pupils to continue in science”. This finding ties in with the results from a study conducted by Sandra Larsen et al., which was looking at short-duration science outreach interventions (4). It is argued that student learning is not the most crucial outcome measure for outreach event programs, even when the topics are aligned with the school´s curriculum. More importantly than assessing students on their learning are identifying changes in their attitude or their awareness of the application sides of science and evaluating how useful students found the event in terms of career advice. The latter is an important criterium, according to a study by Jingoo Kang et al, who found that especially female students decided to leave science-related fields due to a lack of career guidance and not sufficient exposure to the world of science outside school (52).

Barriers to participation of outreach events

Some barriers for schools and students to attend outreach events were mentioned by teachers. Surprisingly only one teacher claimed distance, time and cost for the class to travel to an outreach event as the major hurdles. These were factors studies in the literature considered as the most common event blocker for schools, especially for those situated in the countryside (53). Instead, the teacher survey found that covering for teachers’ absences in schools was the most commonly mentioned hurdle. A few teachers also claimed the limited student places that outreach event organisers have due to the high demand, which was the case for the Quantum Technology School event. Schools were allowed to take a maximum of six students to the event. Besides, teachers stated that missing lessons in schools by students and administering school paperwork to be further barriers to accessing outreach events more frequently.

Outcomes of the follow-up teacher survey

Types of outreach events

The types of outreach events teachers reported to have attended with their class varied from science career fairs, astronomical observatory units, marine education partnerships, to competitive science quizzes. The Big Bang fair was mentioned, which is an annual opportunity for schools to take part in 100 of interactive activities. Students may experience science showcases and speak to a diverse range of STEM employers about possible career options. The astronomical observatory in Edinburgh was mentioned, which regularly organises science talks presenting astronomy and space news. One teacher refereed to an event run by members from the Marine Alliance for Science and Technology for Scotland (MATS), who disseminate their research through talks and activities embracing themes like marine biodiversity, oceanography, and geomorphology. The marine scientists at the outreach event address modern, socially critical questions including how climate variability can impact the ocean’s sustainable resources, or why biodiversity matters in the 21st century. Students have to think critically about the issues modern science is facing. Three teachers had attended the WoSPEG Competition, which is an after-school science competition organised by the West of Scotland Physics Education Group. Teams of three students from different high school across the West of Scotland compete against each other by answering Physics questions taken from the Scottish National Curriculum.

Types of activities

Out of 10 survey responses, the most enjoyed activities reported by students were interactive workshops with 6 votes, as shown in Figure 6. Second were presentations with experimental demonstrations with 3 votes and third individual projects with 1 vote. Presentations without experimental demonstrations and panels of scientists, answering questions by students had also been options in the questionnaire but were shown to be unpopular by students. When comparing these numbers to literature studies, it is unsurprising that interactive workshops and presentations with experimental demonstrations have been chosen first and second as these are also the most frequently occurring type of activities that organisers use to deliver their message at outreach events. Event organiser Susan Arnold Christian, responsible for a large science festival called “Girls Researching Our World (GROW)” at Kansas State University argues that interactive workshops and presentations with experimental demonstrations are not only the most effective activities to involve a large number of students, they have shown to be great at effectively leveraging a small amount of faculty time. Panels of scientists on the other hand use up a significant amount of faculty time, which is why they are not as commonly employed as activities. The nature of the outreach events that teachers had attended with students most likely did not have many panel discussions with scientists, which could be a reason why students did not report them as their favourite activity. Furthermore, research studies in the literature have shown that demonstrations help presenters to capture students’ attention, which could be a reason for presentations without demonstrations to be unpopular amongst students. However, although demonstrations have an enormous power to engage and capture students’ curiosity, science presenters argue that it can be a challenge to manage behaviour, keep students safe, manipulate apparatus and at the same time give a clear, seamless commentary (54). Nevertheless, whenever presenters have the opportunity to “show” science phenomena, whilst also “telling” students about their underlying principles, it is highly recommended to use both as the results show. Experiments are unique to science, and without them, it is more difficult for presenters to bring across their points.

Half of these activities were presented either by a representative from industry (25%) or a representative from a public institution (25%) as shown in Figure 7. Postgraduate students and University lecturers made up each 19%, and Undergraduate and PhD student scientists both 6%. These latter statistics show that the popularity of the presenter does not necessarily depend on the level of experience or achievement in academia. The length of the most popular activities tended to be between 1-3 hours in 70% of the cases, below 1 hour 30% and never more than 3 hours. This is in agreement with literature studies that argue that student attention span is generally no longer than 20 minutes

Attitudes of students

8 out of 13 teachers reported that students most commonly felt excitement before going to an outreach event. The remaining five teachers said that students commonly did not know what to expect. It was those teachers, who replied in the subsequent question that students positively changed their attitude triggered by interesting activities, after the first round of competition, reaching the final in the WoSPEG Competition, and activities using the equipment. The positive changes in attitude that occurred were said to be an increased interest to further study STEM, more motivation towards choosing a career in STEM, and increased enthusiasm towards STEM in the short-term. One teacher noted that it was not a change in attitude, but a change in understanding that students reported after the outreach event.

Findings from Undergraduate survey

The undergraduate survey results showed that the vast majority of the students had not been to an outreach event before coming to University. As shown in Figure 8, 89% of the students indicated that they had never been to any Physics outreach event and only 11% said they had. The types of outreach events students had been to, were the Scottish Wider Access Program (SWAP), the Royal Summer School Edinburgh, the Quantum Technology School (QTS), the Particle Physics Masterclass (PPM), and to "other events", which were science outreach events outside the UK, including one at a science convention centre at the University of Cagliari (Italy), called IDEAS, and one at the Donostia International Physics Centre in Spain, called Top @DIPC.

As depicted in Figure 9, 66% of the students who had been to an outreach event claimed that they had already made up their mind before the event to pursue Physics at University and commented that the outreach event did not have a direct influence to study Physics as they had already decided before. One student, for example, said: “I chose Physics mostly because I enjoy it”. It remains unclear from the comment if the attendance of the outreach event did reinforce their decision or perhaps helped students with their choices within Physics. The remaining 34% commented that the outreach event had an active impact on their decision to study Physics at University. These students made different comments when asked specifically what it was at the event that influenced them most, including:

“I was inspired by the physicists that attended the events, which included a Nobel laureate.”

“Because you get the bigger picture of studying physics and the options you have”

“Particle Physics masterclass was really interesting, so I decided to do joint honours instead of just Maths”

One comment made from a student who had not been to an outreach event was:

“Physics has always been what I wanted since I found that finding out about things has a name”

Findings from Focus groups

The focus groups found that 64% of the students had attended an outreach event and 73% of those had been influenced in their decision to study Physics at University. These results were not in line with the findings from the undergraduate questionnaire, which portrayed a much smaller percentage of students. When students were asked initially who had been to an outreach event, more than half of the students said they had not. However, after it was explained to the focus groups participants what outreach events were, students realized that they had been to at least one and that it had an influence on their decision. The discrepancy in the results of the questionnaire and focus group can be interpreted in several ways. First, it remains unclear if those students who had been to an outreach event were more motivated to join in the focus groups, in which case there was a bias in the focus group data. Could it be that those students, who finished with their task in the lab early and decided to join, were generally more motivated towards science and were thus more likely to have been to an outreach event? What can be said with certainty is that the questionnaire erroneously assumed that all participants knew what outreach events were, which it turned out they did not, which means that a large number of students did not exactly know what the questionnaire was asking for.

Interviews with STEM Ambassadors

Reasons for becoming a STEM Ambassador

To share the passion for science

Everyone mentioned different individual reasons for becoming a STEM Ambassador in the first place, however, the theme that united everyone was the passion to share Physics among others. Regardless of the type of activity or event they were involved in, it was the spirit of sharing, both research and excitement, that was the most commonly found motivation of a STEM Ambassador. One STEM Ambassador explained that she had been a Physics teacher before and enjoyed sharing the passion of the subject to others. She found that the passion for teaching the subject can be lost in the teaching profession when assessment deadlines and strict curricula guidelines need to be followed. "Teaching to the test", is the colloquial term used when teachers direct their instruction toward a specific body of content knowledge or a specific set of cognitive skills represented by a given test (55). Studies performed at the University of Oxford show that students in the UK and the USA are more directed towards learning for exams as opposed to creating a passion for the subject (56). At an outreach event, the focus of teaching lies more on passion-based learning, which STEM Ambassadors can facilitate more easily than teachers as there are no rules or strict guidelines to prepare students to pass an exam. The material at outreach events is cutting-edge science, which sparks students and teachers´ interests.

To fight stereotypes around sciences

Two STEM Ambassadors stated their reason to become a STEM Ambassador was to fight the stereotype that "Physics is a masculine subject". Some literature studies suggest that this stereotype remains a continuing barrier for women to belong in Physics. The origins of perceiving Physics in such a manner can be largely traced back to a child’s development at an early age due to the influence of toys, clothing, or games (57). The STEM Ambassadors both claimed that female toys such as “barbie girls” steered them off from studying science, and it was only because Physics and Maths subjects were mandatory in their senior years in school that they decided to take these subjects further at University. A recent study indicates that Legos and puzzles teach children spatial skills, which help to set the groundwork for further on in their school education when they begin learning about mathematical principals (58). Dolls teach kids empathy and how to take care of another person. Another STEM Ambassador replied that she would like to combat preconceived ideas the public might have about scientists. She said: “I want to put a different face to STEM subjects, away from the image of an old man with grey hair.” Students are unconsciously and involuntarily exposed to the image how a typical scientist should look like from a young age onwards, which potentially influences them concerning which subjects they choose to study and the careers they ultimately pursue. Qualitative observations found in the literature state that science outreach is a step forward in bringing together scientists and the general public in a friendly environment and are a good start to break down the stereotype of the ivory tower (59).

To meet the responsibility every scientist has to share their research with the public

Two STEM Ambassadors said that it´s part of a scientist’s responsibility to explain widely to the public what one is working on. She said: “It should be a core part of our job – not just to do research but to make it publicly accessible.” This statement is viewed critically amongst US scientists, which a survey showed that was released to 4,000 researchers at the American Association for the Advancement of Science (AAAS) annual meeting. 52% of the researchers believed that oversimplifying science to the public was a major problem, especially when engaging via social media to discuss or follow the science. They said they seriously doubted the public´s ability to make the distinction between "good and shaky science". Chelle King, a citizen-scientist coordinator at the Museum of Science in Miami said that the scientific community does not provide much incentive for public communication either (60).

Outcomes for scientists to be a STEM Ambassador

STEM Ambassadors stated that outreach events had substantial benefits for themselves such as gains of teaching or emotional rewards including overcoming the fear of explaining a concept in front of a crowd of people. Younger students said that they had decided to take this role since it was beneficial for their learning to explain more complicated concepts to people of different ages. Two undergraduate students had commented that their outreach so far had helped to consolidate their learning and had elaborated their presentation and rhetorical skills. These qualitative observations tie in with studies found in the literature, which report that STEM Ambassadors described that they were able to understand their science better by explaining it to others with less background knowledge. This social study was conducted by researchers at the University of Nottingham, analysing qualitative feedback forms given out to presenters after a science outreach event that had reached out to 1000 local school students (61). Also, many other positive outcomes have been reported that include growth in confidence, especially after STEM ambassadors had been presenting at several outreach events and personal career gains that include clarification and confirmation of career plans or new career ideas based on the STEM Ambassador experience (4).

Outcomes for teachers to come to an outreach event

Obtaining a different approach to teaching

STEM Ambassadors reported that teacher’s ´ feedback was that they did not only learn new facts, but also new ways of viewing and approaching scientific problems after attending their outreach event. One STEM Ambassador had commented that it´s nice for teachers to listen to someone else’s perspective and view a different approach to a Physics question. Another STEM Ambassador stated that outreach events could help teachers be able to teach particular parts of the curriculum in a different, perhaps more modern way. Working with STEM Ambassadors may offer teachers opportunities associated with Continuous Professional Development in keeping up with forefront science, ideas to incorporate into their teaching or additional information they can use to pursue further with their students. Since September 2019, Scotland has adopted the Curriculum for Excellence, which includes an area called topical science. In this area of the curriculum, the teachers are requested to bring in real, cutting edge scientists into the classroom. A comment by a STEM Ambassador was:

“Teachers working with scientists that are doing current research, are well-placed to bring in that part of the curriculum. Those teachers who do not have the expertise in a specific area can get support from scientists, who are specialists in the field.”

This qualitative observation is in line with a similar study from the literature that identified that teachers valued STEM Ambassador scientific expertise and the relevance of the topics they choose.

A break from routine

“Seeing someone researching your field reminds you, how mind-blowing and fascinating your field is”, commented a STEM Ambassador, who is a PhD student in Gravitational Wave Research. “This applies to any job. Sometimes as a teacher, you might feel bored after teaching the same topic over and over again. Listening to someone else talk about their passion for the topic might reinvigorate your passion for the subject, too.” Research by the Association of Teachers and Lecturers found that two-thirds of teachers in the UK have considered leaving their profession (62). Those teachers who had been in the profession for more than 10 years had said they experienced a lack of enthusiasm after teaching the same subject, grade level, and lessons. Teachers might inevitably lose motivation for the subject over time, especially when students are not connecting to what teachers are teaching them (63). Hence, seeing real-world applications at outreach events can be highly valuable to remind them that what they are teaching is interesting and can give them real examples to use during their lessons.

Outcomes for students to come to an outreach event

A change in attitude towards science

STEM Ambassadors stated that presenting the material from a currently active presenter´s view gives students an insight how to think like a Physicist, a Biologist or a Chemist rather than learning facts for the exam, which is a major improvement in student´s attitude. Students have the advantage of learning the very latest cutting-edge science from researchers at outreach events, which makes students realize that many questions in science have not been fully explored yet. One STEM Ambassador claimed: "Students´ attitudes towards Physics change massively when they see current research that is alive rather than being a subject that has answers to questions from a bunch of dead guys." Furthermore, STEM Ambassadors stated that their enthusiasm towards science sparked students´ interests and positively influenced students ‘attitude. “It´s wonderful for students when they can experiment with things in a new learning environment”, was commented by a STEM Ambassador. The interviews further found that the perception students have of their learning environment linked to their attitude. This finding is in line with a literature study that gathered data from 35 STEM ambassadors, who had been involved in teaching 12 outreach events across the US and the Netherlands (64). The study found that students´ perceptions of the learning environment at the outreach event positively correlated with the attitude they had towards STEM, and with their motivation to engage in STEM learning.

A different presenter

In line with the literature, the interviews with the STEM Ambassadors found that presenters can give students a different perspective than their teachers. Their approach towards students can be more on a peer level. One STEM Ambassador said: “Having a discussion with students rather than lecturing at them, can be massively beneficial.” In addition to the novelty of the presenter being different from a regular teacher, STEM Ambassadors noted that they were regarded as role models with different appearances and individual personalities. It would be easier for students to identify themselves with presenters, when they were separated by a small age difference or when being of the same gender. A comment by a STEM Ambassador was: "Regardless of the type of event, if students speak to a scientist's face to face, they start noticing that they are humans as well, and they can be role models. If it is a woman, and she engages with young girls that can break down stereotypical barriers." As students participate in different STEM outreach events, they will encounter different people in the STEM field and students will realize that STEM Ambassadors are not all the same. They can come from all kinds of different backgrounds and demographics. One STEM Ambassador said: “You have to see it, to be it.”

Science outside the classroom

For students, learning about science at a different venue can be more inspiring and meaningful than in an ordinary classroom. This view is shared amongst all STEM Ambassador interviewees and is supported by literature sources (65). Classroom teaching is often more theoretical while practising science outside the classroom in a realistic work environment adds a sense of occasion to learning and makes a lesson more memorable. Students develop the ability to tackle real-life problems and benefit from learning experiences beyond science. One STEM Ambassador explained: "When students are coming to an event that is held at a University, for example, it can break down barriers such as the fear of leaving home and going to an unfamiliar place. They feel more comfortable after."

Age at which students benefit most from an outreach event

Two stages are important: at the primary school level to encourage students from early onwards and at secondary school level at the S3 level, when students are confronted with subject choices

Evaluation of outreach events

The consensus amongst STEM Ambassadors was that evaluation of outreach events is important and critical for improving individual activities of outreach programs. However, none of the STEM Ambassadors except one had heard of any of the scientific evaluation methods mentioned in this report, such as the Logic Model, the Theory of Change or the Kirkpatrick model. However, all STEM Ambassadors said that they evaluated to some extent either through informal techniques such as by show-off hands or more formal research methods such as using questionnaires. No STEM Ambassador reported using a formalised document. One STEM Ambassador said: "Every event is different. Sometimes I work as an individual, sometimes like a collection of STEM Ambassadors, sometimes as part of a research group. Hence, it is very difficult to create a blank evaluation sheet”. Another STEM Ambassador referred to the use of a technique called: "a postcard to a future self”. This involves asking students to write down on a postcard their address and one thing that they will change in their life, which they took away from the event. A couple of months later, the event organiser posts the postcard to the participants. They are faced with what they said they would do and then self-evaluate if they have done that thing. Event organisers have to decide a sensible period to send it back. After they have sent the postcard, they would recontact the participants again via email to evaluate if people did what they said they were going to do. The same STEM Ambassador also reported using the “Magic moment book”, which is a book that keeps track of any anecdotes about the long-term impact science volunteers had in their lives. Every time an anecdote is reported, it needs to be written down to make sure it is recorded for longer-term case studies.

Discussion of the results

Limitations of the study

Initial Teacher survey and follow-up teacher survey

The results from the initial and follow-up survey are expected to be affected by the Priming Effect, which refers to the psychological phenomenon to rate an activity or event more positively the nearer in time they are asked about it (also see Section 4.1 Limitations of social research evaluations). The initial survey is likely to be rated more positively as participants were asked immediately after the outreach event. It is difficult to compare the two surveys as the questions, the audience and the sample size of both were different. According to the Priming Effect, it would also be expected that those activities that came last at the event to be rated the most enjoyable. However, this was not the case as the last activity at the QTS was a presentation without experimental demonstrations, which was rated to be unpopular amongst students. The groups of students who participated in the outreach events did not represent the whole student population, which is selection bias and a further limitation of the social study. The first survey asked teachers about the attitudes, decisions and thoughts of those students, who had been pre-selected. Not all students from the class could be brought to the outreach event due to the limited places available. The majority of students were those, who had been motivated in class, and who had shown genuine interest in going to the event. The teacher's responses were based on able and motivated students, who were likely to be enthusiastic about science subjects in the first place. Incorrectly excluding less motivated students leads to skewed data results. It is not fair to draw general conclusions about all the student population based on the teacher survey. Both surveys had a sample size of below 15, which is relatively small considering all science teachers in Scotland. A small sample size reduces the statistical power of the study and increases the margin of error, which leads to less conclusive and meaningful results.

Survey and focus groups of Physics 1 Undergraduate

The Physics 1 students who participated in the survey did not all know the meaning of outreach events, which was erroneously assumed when designing the questionnaire.

Interviews with STEM Ambassadors

Using social media platforms to promote a study introduces a demographic selection bias by recruiting only those who are active social media users and friends or followers of the person, who posted the study. Besides, the more internet-active population tends to be younger and excludes those countries or regions that are restricted in access to social media.

Discussion and recommendations based on future outreach events

Effectiveness of outreach interventions to attract students into sciences

Outreach programs have three main goals which are; improvement of learning, promotion of civic engagement and the strengthening of communities through the addressing of the needs of societies. Based on the results of this research study, there is supporting evidence for science outreach events at being effective at attracting students into sciences. Teachers surveys and STEM Ambassadors interviews demonstrated among other things that science outreach events have a positive impact on students learning, their attitude, and towards reducing stereotypes. Outreach programs provide students with opportunities for understanding the contents of their courses, making them aware of different scientific concepts. Students reported their enthusiasm for training in an outreach program with a sense of growing confidence in their abilities and the additional development of their theoretical scientific knowledge. The students further recognised the value of a multi-disciplinary approach to science and reported feeling comfortable working with teachers from different institutions. According to the students, this helped them with defining their personal weaknesses and strengths. This came about as a result of working with supervisors who accorded them more freedom and also gave them opportunities of making decisions on their own. Throughout the program, it also dawned on the students that learning within an outreach context is not as neatly pre-packaged and structured with proper time schedules as they were used to in their normal school routines. Being involved in the outreach program helped boost the self-confidence of the students as they were allowed to engage themselves entirely in the management of those duties that had been assigned to them.

The presenter

The research showed that the presence of a female presenter alone is insufficient to change the perception that students have that science subjects are for men only. To tackle this stereotype, a talk or an informal discussion with students at the beginning of the outreach event could be beneficial. The consensus among STEM Ambassadors was that the presenter´s age and gender played an important role for students to identify themselves with the presenter. When STEM Ambassadors are a lot older, they are seen more like a figure of authority. However, the follow-up teacher survey entirely disagreed with the outcomes of the STEM Ambassador interviews in this respect. Teachers believed that students cared about neither gender nor age of the presenter. Instead, the results showed that the quality or competence of the presenter was crucial.

The proposal of a common standard to evaluate outreach events

To ensure consistency in the evaluation and allow for comparisons between outreach events, a common evaluation standard was proposed by several literature studies. However, based on the STEM Ambassador interviews, it was found that a formalised document would not be helpful as every event is different. It is difficult to create a blank evaluation sheet, as the requirements vary too much. Perhaps, categorical evaluation sheets would be possible by classifying outreach events according to cost, time and expertise.

Recommendations for Quantum Technology School in 2020

To promote greater diversity in the intake into Physics at the University of Glasgow, a more diverse group of students need to come to the outreach events. These include female students, students from lower socio-economic classes and non-white ethnicities. Drawing on the theory, discussed in Section 3 of this report, the following logic model could be used to plan for the QTS event taking place in September 2020:

The long-term goal of the QTS event in 2020 is to promote greater diversity among the first-year Physics intake students at the University of Glasgow. To further elaborate the planning, a Theory of Change model (see Figure 11) and a Kirkpatrick Model (see Figure 12) could be applied to contribute to the overall long-term goal and simplify the evaluation process by measuring if short, and medium-term goals have been achieved.

Conclusion

This report investigated the impact of science outreach events targeted at school students. Different literature sources were reviewed and analysed to find existing assessment strategies that social researchers use to evaluate outreach events. The Theory of Change and the Logic Model are the most commonly employed models to graphically illustrate how resources and activities can lead to short, medium and long-term objectives. A combination of quantitative and mainly qualitative research was used to explore the effects of outreach events through feedback from three different groups: high school Physics teachers, STEM Ambassadors and first-year undergraduate students. Based on this research, there was supporting evidence to suggest that science outreach events are being effective at attracting students into sciences. Teacher surveys and STEM Ambassador interviews showed that science outreach events have a positive impact on students learning, their attitude, and towards combatting stereotypes that individuals might have about science. Outreach programs have numerous benefits in that they provide students with academic foundations and also help in the development of their self-confidence, responsibility towards the community and also leadership qualities. As such, it is proven that outreach programs have an impact on the participant’s personal development, academic development and civic responsibility. The initial and follow-up teacher survey found that outreach events are valuable and beneficial to students as they learn science outside the classroom in an informal learning environment. The undergraduate survey results indicated that 89% of the students had never been to any Physics outreach event before coming to University. Yet, the focus groups revealed that the majority of students did not know the definition of outreach events. After explaining the meaning to focus group interviewees, 64% of the students said that they had attended an outreach event and 73% of those had been influenced in their decision to study Physics at University. The follow-up teacher survey discovered that the most enjoyed activities reported by students were interactive workshops with 60%, presentations with experimental demonstrations with 30% and third individual projects with 10%. Furthermore, there is strong evidence that outreach events help teachers to reignite their passion for the subject, which has positive effects on their classroom practise. STEM Ambassadors stated that outreach events had substantial benefits for themselves such as gains of teaching or emotional rewards including overcoming the fear of explaining a concept in front of a crowd of people.

References

Ngumbi, Esther. Scientists Should Talk Directly to the Public. Champaign : Scientific American, 2017.

Laursen, Sandra. What Good Is a Scientist in the Classroom? Participant Outcomes and Program Design Features for a Short-Duration Science Outreach Intervention in K–12 Classrooms. Boulder : PMC, 2007.

Burke, James. The Impact of science on society . Washington, D.C. : NASA, 1936.

Has Feminsim Changed Physics? Bug, Amy. 3, s.l. : Spring, 2003, Vol. 28.

Glasgow, University of. Scottish Funding Council: Gender Action Plan. Glasgow : University of Glasgow, 2018.

University-led STEM outreach programs: purposes, impacts, stakeholder needs and institutional support at nine Australian universities. Sadler, Kirsten. s.l. : Studies in Higher Education, 2018.

Integrating Research and Education with Public Outreach at Coastal Laboratories. Milliman, John D. s.l. : Biological Bulletin, 1996, Vol. 190. 10.2307/1542550.

Friedman, Alan J. Report from a National Science Foundation Workshop. Alexandria : The National Science Foundation, 2008.

Hillier, Emily. An Evaluation of Ipad Implementation Across A Network of Primary Schools in Cardiff. Cardiff : School of Education Cardiff Metroplitian University, 2014.

Irving, Donna. Teaching and Learning in Action. s.l. : Health Libraries Group, 2019.

Waldman, Caroline. Four Elements for Creating a Positive Learning Environment. Washington D.Cc : Alliance for Excellent Education, 2016.

Weiss, Carol Hirscbon. Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families. Washington D.C. : Initiatives, 1995.

Jim McCambridge, John Witton, Diana R. Elbourne. Systematic review of the Hawthorne effect: New concepts are needed to study research participation effects. London : PubMedCentre, 2014.

Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Kruger, Justin,Dunning, David. Ithaca, New York : Journal of Personality and Social Psychology, 1999, Vol. 77 (6).

Examining Factors Impacting Online Survey Response Rates in Educational Research: Perceptions of Graduate Students . Bista, Amany Saleh and Krishna. 29, s.l. : Journal of Multi-Disciplinary Evaluation, 2017, Vol. 12.

Science learning through scouting: an understudied context for informal science education. Jarman. 4, s.l. : International Journal of Science Education, 20122, Vol. 27.

Behrendt, Marc. A Review of Research on School Field Trips and Their Value in Education. Ohio : International Journal of Environmental & Science Education, 2014.

Gender study on the relationships between science interest and future career perspectives. Kang, Jingoo. 1, Joensuu : International Journal of Science Education, 2018, Vol. 41.

Underachievement in physics: When intelligent girls fail. Sarah I. Hofer, Elsbeth Stern. Zurich : Institue for research on Learning and Instruction, ETH Zurich, 2016.

“She Won’t Make Me Feel Dumb”: Identity Threat in a Male Dominated Field. Hirshfield, Laura E. Michigan : International Journal of Gender, Sience and Technology, 2010.

Braund, Martin. Student attitudes, engagement and participation in STEM subjects. York : The University of York, 2013.