Universe in the Classroom Phase 1: Final Report

Written by Sarah Roberts and Haley Gomez

1

Chapter 1: Universe in the Classroom 1.1 The National Context 1.2 What is Universe in the Classroom (UitC)?

Resources provided as part of UitC Stellar Role Models (STARS) 1.3 Impact figures 1.4 Key Findings of Universe in the Classroom 1.5 Recommendations for future projects Chapter 2: Who Did Universe in the Classroom Reach?

1. Free School Meals 2. Below Average Attainment 3. Annual Budget per Students 4. Welsh­speaking Schools 5. Geographical Diversity 6. Special Schools 2.1 Science CPD, Support and Advice to Schools through UitC Chapter 3: Results and Findings from Universe in the Classroom 3.1 The Teaching environment 3.2 Did UitC improve STEM in Welsh primary schools?

3.3 Did UitC provide additional support and training for teachers?

3.4 Did UitC challenge pupils’ perceptions of STEM? Figure 17: The four most common features on primary students’ drawings of scientists create pre (left) and post (right) STARS workshop.

3.5: Did UitC provide challenging learning opportunities for primary school pupils?

Conclusions Appendix A ­ Resources used in UitC A1. Universe­in­a­Box A2. Earth Ball and Activity Book A3. Cosmos in your Pocket A4. “Where is Spacey’s Planet?”

A5. Robotic Telescope for Wales Appendix B: Evaluation of UitC Training and Resources B1. Pre­teacher training survey B2. Post­teacher training evaluation survey B3. Evaluation collected from pupils B4. Draw a Scientist examples: B5. Personal Meaning Maps (PMM) example: B6. PMM Complex and Elementary concept list

2

3

Chapter 1: Universe in the Classroom 1.1 The National Context

In 2012, Wales scored lower than all other UK nations for Science in the international PISA1 study, this score is lower than the Organisation for Economic Co­operation and Development (OECD) average and has been falling since 2009. A recent Estyn thematic report (Science in Key Stage 2 and 32 ) recommended the following to improve primary level engagement with STEM: ● provide challenging science opportunities to stretch all pupils, particularly the more able, and eliminate tasks that are too easy ● provide more opportunities for pupils to pursue their own scientific interests ● provide training for teachers with weak science subject knowledge ● provide science continued professional development (CPD) ● ​provide more professional development, support and advice to schools on science teaching and learning.

Beyond STEM subjects, thematic reports from Estyn identified other global concern areas for Wales including: ”tackling aspiration, engagement, emotional skills, well­being and social mobility in schools in challenging circumstances (e.g. poverty, lack of engagement, lack of positive role models).”

More recently, the Institute of Physics published astonishing results3 that schools are extremely influential in determining whether or not girls go on to study science beyond secondary level, with half of schools not sending any girls on to study A Level Physics in England. A similar (but worse) picture also exists in Wales (David Cunnah, ​private communication​), despite girls performing just as well as boys. The lack of support by lack of scientifically­aware teachers4 and students’ perception of STEM is crucial in perpetuating this gender bias in physics. Furthermore, there is a lack of positive male role models in primary schools, with one school in four not having access to male teachers in Wales.

In this landscape, Universe in the Classroom (UitC) was developed in partnership with Cardiff University, non­profit organisation Las Cumbres Observatory Global Telescope Network5 and the award­winning, pan­European Universe Awareness6 project. UitC is designed to bring the 1

PISA is the Programme for International Student Assessment (PISA), a worldwide study by the Organisation for Economic Co­operation and Development (OECD) in member and non­member nations of 15­year­old school pupils' scholastic performance on mathematics, science, and reading. The results from 2012 were from a comparison against 65 nations.

2 ​http://www.estyn.gov.uk/english/docViewer/281872.5/science­in­key­stages­2­and­3­june­2013/?documents=date 3 http://www.iop.org/education/teacher/support/girls_physics/file_58196.pdf 4 http://www.iop.org/publications/iop/2013/closingdoors/ 5 lcogt.net 6 unawe.org

4

unique inspirational qualities of astronomy to welsh primary schools to improve STEM (science, technology, engineering a raise primary school’ teacher participation in STEM activities all over Wales and introduce challenging learning opportunities for students.

Particular emphasis was placed on raising social mobility, engagement and STEM skills in unengaged schools and underserved communities. The project also facilitated engagement between Welsh scientists and local communities by building a diverse team of scientists to run school workshops through an undergraduate role model scheme.

In this report we will discuss the details and evaluation of the Universe in the Classroom project including its implementation and impact. As part of this project we also recommend the following points for future Welsh Government projects working with primary schools.

1.2 What is Universe in the Classroom (UitC)?

UitC ran from Jan 2014­March 2015, worked with ​99 schools​, 131 teachers​, ​34 undergraduate and postgraduate ​role models, and indirectly reached more than ​6000 school children at Key Stage 2 in this period. Specifically, we achieved the following aims and deliverables:

Aims

Delivered

Deliver ​5­7 CPD sessions for a total of ​70 primary school teachers at Cardiff University.

7 CPD sessions in Cardiff University and ​1 CPD training in Machynlleth (mid­Wales) for a total of ​56 primary school teachers.

Deliver ​30 ‘twilight’ CPD sessions for teachers/schools in­situ (allowing the training to be conducted further afield than the Cardiff base), which is particularly important for rural schools.

5 ‘twilight’ CPD sessions for teachers/schools in­situ due to the preference of teachers to attend the CPD days.

Train a team of ​10­20 undergraduate ​astronomy students representing a diverse range of backgrounds to deliver the CPD with leadership from a member of the core team.

Created a team of ​34 undergraduate and postgraduate ​astronomy, physics and mathematics students representing a diverse range of backgrounds, different genders, STEM subjects and levels of education, to deliver workshops for primary school children. Being part of UitC improved their transferable and employability skills.

Deliver Universe­in­a­Box and Robotic Telescope for Wales workshops for primary students in ​30 schools ​(delivered by trained undergraduates and staff).

21 Universe­in­a­Box and Robotic Telescope for Wales workshops for ​871 ​primary students (delivered by trained undergraduate and postgraduate role models).

Resources provided as part of UitC Each school received the following resources to take back to their schools: Universe­in­a­Box toolkit, Earth ball and activity booklet, Cosmos in your Pocket activity books (available in 5

Welsh language), ‘Where is Spacey’s Planet?’ storybook and access to the LCOGT robotic telescope network. The teachers were trained in using the resources and robotic telescopes in their school.

Stellar Role Models (STARS) To carry out the ambitious goals of Universe in the Classroom, Cardiff University undergraduate and postgraduate students studying astronomy, physics, mathematics or engineering were invited to volunteer as Stellar Role Models (STARS) to help organise and run primary school workshops designed to inspire and educate children and demonstrate to teachers how the Universe in the Classroom resources can be used with their class.

By opening the project to STEM students we could schools offer a diverse pool of representatives; including Welsh­speakers, female scientists and a range of ethnic groups and backgrounds, who aimed to tackle diversity and gender issues in science and technology across Wales and broaden young children’s ideas of who can be a scientist at an age where impressions can become fixed.

Stellar Role Models were required to attend a training sessions on outreach, education and the project resources before being given the opportunity to work with local communities across Wales, making new friends, helping improve STEM skill capacity for the future of Wales, helping improve Welsh education of STEM and motivating young children to learn (and develop a life­long enjoyment of learning).

The scheme was not only beneficial to the community, the students also gained valuable communication, presentation and education skills, improving their employability. 34 students volunteered to take part in the scheme and 21 STARS workshops were held in schools for 871 Welsh primary school children. The STARS scheme aimed to directly educate and inspire students as well as addressing any negative connotations the children may hold about STEM by demonstrating that scientists are not confined to a small stereotype, in fact, anyone can be a scientist.

1.3 Impact figures

At the end of the Universe in the Classroom project in March 2015, the impact figures were as follows: ★ Universe­in­a­Boxes delivered to schools: ​99 ★ Teachers trained: ​131 ★ Teacher CPD and twilight completed: ​8 ★ Teacher twilight completed: ​5 ★ STARS trained: ​29 ★ STARS school workshops completed: ​21 ★ Children directly reached by STARS workshops:​ 871 6

7

1.4 Key Findings of Universe in the Classroom

1. A large number of primary schools located in mid and north­west Wales do not accept correspondence in English language only.

2. Teachers expressed a preference for full­day CPD workshops hosted during school time over post­school twilight CPD sessions, despite the need to organise substitute cover for their classes 3. We found that an unacceptably large number of Welsh primary school teachers assign just one hour per week to teaching Science; ​89%​ of the primary teachers who participated in this project stated that they spent less than 2 hours per week on science in their classrooms.

4. Before this project, ​1% ​and ​4% of the teachers gave correct answers to questions on lunar phases and seasons consecutively. After our CPD this increased to ​20%​.

5. 94% of teachers used our resources in their classroom within 6 weeks after our CPD with ​55%​ sharing them with other teachers in their school.

6. 97% of teachers stated that they felt more confident teaching STEM after our training/workshop events.

7. Before participating in UitC, ​85% teachers stated their schools did not have the resources necessary to carry out STEM activities in the classroom. After UitC ​70% claimed their schools would benefit from a second Universe­in­a­Box toolkit.

8. Student evaluation shows that children as young as 7 years already appeared to hold preconceptions about scientists. However, children who experienced a STARS workshop changed their perception of scientists from something superficial (eg someone wearing glasses) to someone who performs experiments and finds answers.

9. Before starting this project both male and female children were more likely to draw a male scientist than a female scientist.

However, the number of children who abandoned the male stereotype by drawing a female scientist increased by ​3% when a female STAR hosted the school workshop. A more noticeable result was that more boys and girls were ​10% more likely to draw a female scientist if they were taught by a female teacher.

10. We found a demonstrably large increase in the complexity of the childrens’ vocabulary and perception of space concepts after attending our workshops.

8

1.5 Recommendations for future projects 1. Teachers are extremely busy and schools should be approached at least a year in advance to organise twilight session, this is only possible if teacher training projects such as UitC are funded to run for longer than 14 months.

2. Welsh translations of materials and communication with schools is crucial.

3. We suggest that Welsh primary schools should have a target to schedule at least 3 hours of science lessons per week, as is commonly seen at Key Stage 3. This requires further support and guidance from the Welsh Government.

4. The collection of evaluation data post­teacher training workshop is essential to monitor the use of resources, for insight into the impact of the programme and to provide feedback for any necessary improvements. However, motivating teachers to return evaluation forms proved difficult. We recommend including incentives such as competition prizes or additional teaching kits to improve take up and engagement.

5. Organise longer, in­depth training 1­day workshops to address teachers’ needs. Any outreach project should be prepared to supply the cover charge required for the teacher to be out of the classroom during this period.

6. Due to awkward transport routes across Wales, several teachers from rural north and west Wales expressed a preference to attend CPD events held in Cardiff rather than events held closer in some rural areas closer to their school. This should be taken into consideration when selecting schools to invite to events in future.

9

Chapter 2: Who Did Universe in the Classroom Reach?

Schools were hand­picked to maximise the impact of Universe in the Classroom on unengaged schools, based on the following defined criteria: 1.

Schools with high percentage of children living in relative poverty based on the percentage of students receiving free school meals, 2.

Schools with below­average levels of attainment in Science at Key Stage 2, 3.

Schools with a low annual budget per student, 4.

Welsh Medium schools, 5.

Geographical Diversity (north and west Wales), 6.

Special Schools.

A breakdown of the 98 schools who participated in UitC based on these criteria is shown in Figure 1 (Table 1), with detailed breakdown in Table 2.

Figure 1: Universe in the Classroom participant schools (FSM = free school meals) based on selection criteria. ​Left: ​The majority of schools in the project can be classified as eligible based on two or more of the five defined criteria stated at the beginning of this section. ​Right: ​Additional information about the schools that are eligible for the project based on two or more of the defined criteria listed in left figure. ​The data for this plot are shown in Tables 1 and 2 with colour scheme matching row colours.

Table 1​: Breakdown of Universe in the Classroom participant schools based on selection criteria as shown in ​Figure 1​. The majority of schools can be classified as eligible for the project based on two or more of the six defined criteria.

Demographic

Number of participant schools (%)

Schools with high percentage of students eligible for Free School Meals (>30%)

49.5%

10

Schools located in remote regions (north and west Wales)

30.3%

Schools with low percentage of students achieving expected Level 4 grade at KS2 (<60%)

36.4%

State schools receiving a low annual budget per student (<£3000)

45.5%

Welsh Medium Schools

22.2%

Special Schools

6%

Table 2​: Breakdown of the demographic of Universe in the Classroom participant schools as shown in Figure 2​.

Demographic

Number Schools

Schools with high percentage of students eligible for Free School Meals (>30%)

6

Plus

Low attainment levels

11

Low budget per student

5

Welsh Medium

1

Remote region

2

Remote region and low attainment levels

2

Remote region and Welsh Medium

1

Low attainment levels and low budget

21

Schools located in remote regions Welsh Medium

17

Low budget per student

1

Welsh Medium schools

1 Low budget per student

Low budget per student (<£3000) Plus Special Schools

Participant

7

Plus

Plus

of

2 14

Low attainment levels

2 6

11

1. Free School Meals A 2013 7 study by the Welsh Government’s Statistics for Wales found that primary school pupils eligible for free school meals (FSM) performed less well than their non­eligible counterparts at both Foundation Phase (FP) and Key Stage 2 (KS2). A lower percentage of FSM­eligible pupils achieved the expected level or above (Outcome 5 at FP and Level 4 at KS2) across the board in English or Welsh (first language), Mathematics and Science. This study appears to provide clear evidence that relative poverty (based on FSM) appears to have a notable affect on a child’s educational performance, and 17% of children in Wales are eligible for FSM8 .

Universe in the Classroom provided free STEM resources (Universe in the box) and free teacher training and workshops to 49 schools with an above average percentage of pupils eligible for FSM. By targeting these schools, the project aims to raise attainment levels in Wales from Foundation Phase through to Key Stage 2 and beyond.

2. Below Average Attainment All learners in their final year of Foundation Phase and Key Stage 2 must be assessed through teacher assessments. The general expectation is that the majority of 7 year olds will attain outcome 5 in each area of learning at Foundation Phase and the majority of 11 year olds will attain level 4 in each subject at the end of Key Stage 2. The percentage of students in Welsh primary schools achieving the expected level or above at Foundation Phase (Outcome 5) was 83% and at Key Stage 2 (Level 4) was 84.3% in 20139 .

Universe in the Classroom targeted 36 welsh primary schools achieving below the average standard with the aim of improving attainment levels in these schools across the board.

3. Annual Budget per Students Welsh primary schools receiving below the average budget per student for the 2013/2014 academic year were identified using data provided by the Royal Statistical Society, collated from a range of sources including the Pupil Level Annual School Census (PLASC), National Data Collection (NDC), awarding organisations such as AQA, EDEXCEL and OCR, and directly from local authorities (LA). Evaluation carried out during our teacher training workshops (Section 3.1) also showed that Science is prioritised behind English, Mathematics and Welsh in primary school classrooms, suggesting that schools with limited budgets would focus their resources on these subjects before science.

Schools with the lowest budget per student were targeted by UitC as they have less means to acquire educational materials and were likely to benefit from the free resources. Almost half 7

​http://dera.ioe.ac.uk/19260/1/140122­academic­achievement­free­school­meals­en.pdf ​http://www.jrf.org.uk/sites/files/jrf/wales­education­poverty­summary.pdf 9 https://statswales.wales.gov.uk/Catalogue/Education­and­Skills/Schools­and­Teachers/Examinations­and­Assessme nts/Foundation­Phase 8

12

the schools (45) participating in UtiC fit into this category, and 31 of these schools were also suitable candidates for the project based on other criteria detailed in this section (e.g. 21 were also high FSM ​and​ low­attainment schools).

4. Welsh-speaking Schools Almost a quarter of welsh primary school students attend Welsh Medium schools10 (22%).

Similarly almost a quarter of schools participating in Universe in the Classroom are Welsh Medium (22). These schools consistently achieve higher grades and better inspectorate reports than English Medium schools, however the lack of educational resources available in welsh language and distance from working scientists who cluster around large cities classifies these schools as underserved.

Many national science projects are in the English language and do not cater to Welsh speakers. UtiC is primarily an English­language project, however we do provide some resources in Welsh language (see Appendix: ​A3​). Additionally, the UtiC resources are comprised largely of hands­on activities and materials that do not require students to read or write in English.

Feedback provided by participating teachers (Section 3.2) showed that the Universe­in­a­Box toolkit was used between 1 and 8 times per Welsh Medium school within the first 6 weeks suggesting these resources were popular in all schools.

5. Geographical Diversity To maximise the reach of Universe in the Classroom and ensure that schools outside of the populous south Wales region were offered the opportunity to participate and benefit from the project​, ​primary schools from each of the 22 Welsh counties were invited to participate in training events and school workshops. These schools were selected using our previously defined criteria (FSM, Welsh Medium etc). The locations of the final schools who participated in UitC is shown in the left image of Figure 3. The schools cluster around south east Wales and the northern border, with only 2 counties (Powys and Denbighshire) having no schools sign up to our project. The population density of Wales is shown in the right hand side of Figure 3, where it becomes clear that the UitC schools trace the most populated Welsh areas.

6. Special Schools Of 99 Universe­in­a­Box toolkits, 6% were handed to special schools in south­west Wales.

These schools actively contacted our project after hearing about us through teaching networks. This demonstrated significant enthusiasm by teachers in these schools. These schools fit into our chosen demographic of underserved and unengaged welsh primary schools.

10

​http://gov.wales/docs/statistics/2012/120711sdr1082012en.pdf

13

Figure 3​: On the left is a map showing the distribution of Universe in the Classroom participant schools. On the right is a Wales population density map based on data from StatsWales and Ordnance Survey data.

2.1 Science CPD, Support and Advice to Schools through UitC

Selected schools were invited to send one of their teachers to a training event during which they would receive a selection of free STEM education resources for their school, including a Universe­in­a­Box ​hands­on education toolkit, an Earth ball and accompanying activity book, an inspirational storybook called ​‘​Where is Spacey’s Planet?​’ and some ​Cosmos in your Pocket activity books.

Figure 4 Left: a map of Wales showing the distribution of CPD workshops hosted by Universe in the Classroom.

Universe in the Classroom trained 131 welsh primary school teachers through a mixture of full­day CPD workshops and shorter twilight training sessions held in­situ. The training sessions provided an opportunity to demonstrate how the activities can be linked to the 14

national curriculum and integrated with other subjects, improving the likelihood of teachers using the resources use in the classroom. Teachers were also encouraged to ask any questions about the activities, resources and necessary background science, and invited to discuss best practice with other experienced educators in attendance.

During the project, 8 full­day CPD sessions were hosted, in south wales and mid­Wales along with 5 twilight sessions which took place in south, north and west Wales (Figure 4).

15

Chapter 3: Results and Findings from Universe in the Classroom 3.1 The Teaching environment

At the start of each training workshop evaluation was collected from teachers in the form of a pre­workshop survey (Appendix: ​B1​). The survey collected data on a range of issues, including how confident teachers were when teaching STEM subjects in their classroom; 56% rated their confidence as 3­out­of­5 prior to the workshop, with just 7% rating themselves as 5­out­of­5 (Figure 5).

Figure 5​: Teacher responses to the pre­evaluation question, “How confident are you when it comes to teaching science?” 5 = very confident, 1 = very uncertain.

Surprisingly 89% of teachers said that they only committed two hours of lesson time (or less) to science per week. The low number of hours spent on science is likely linked to the low priority ranking of the subject as prioritised by most teachers (Figure 6). 88% teachers placed science 3rd or 4th priority, behind english, maths and welsh language in their classroom, only 8% placed it at above 3rd priority and some teachers place it as low as 6th place compared to other subjects.

Figure 6: How teachers prioritise Science compared to other classroom subjects (1 is high priority and 6 is low) based on pre­workshop evaluation.

16

When teachers were asked how they plan their lessons, the result was a close split between teachers planning their lessons based on the National Curriculum and those planning them based on School Strategy (Figure 7 left). Finally, teachers were asked whether they felt their school had the resources necessary to carry out hands­on science activities and 85% teachers felt there was a distinct lack of resources in their school (Figure 7 right).

Figure 7​: Left: How teachers prioritise subjects taught in their classroom based on teacher responses to the pre­workshop evaluation. Right: whether teachers felt their school had access to the resources necessary to carry out STEM activities.

Each teacher was sent a follow­up survey six weeks after they attended a Universe in the Classroom training workshop. The survey was completed by 31 teachers (24% of our participants).

3.2 Did UitC improve STEM in Welsh primary schools?

Figure 8 (left) shows that 97% of teachers felt ​more confident teaching STEM in their classrooms after attending a Universe in the Classroom teacher training workshop. Teachers also used the resources in multiple subjects outside of STEM (94% of teachers had used the resources to teach 8 subjects ­ Figure 8 right). Overwhelmingly the Universe in a box kit was seen as the most useful (Figure 9) with 70% of teachers agreeing. Teachers also shared the resources with other staff in their school (55% had already shared the resources with at least one other member of staff) and 70% claimed that a second Universe in a box toolkit would be useful.

17

Figure 8​: ​Left​: teacher responses to the post­evaluation question, “Did the Universe in the Classroom training session make you feel more confident when teaching science?” 97% answered positively with just one teacher stating that the training did not improve their STEM confidence. ​Right​: The number of teachers who used the Universe in the Classroom resources to teacher each subject in their classroom.

Figure 9: The resources each teacher found most useful for carrying out STEM activities in their classroom.

3.3 Did UitC provide additional support and training for teachers?

The pre­and­post workshop evaluation also assessed the participants knowledge gain on two common primary­level astronomy topics: the Seasons and Lunar Phases. The evaluation surveys included a multiple choice question for each topic (see Appendix: ​B​).

Pre­workshop just 1% of teachers answered the Lunar Phases question correctly with 4% of teachers answering the Seasons question correctly. Post­training there was a clear improvement in knowledge of these topics; the number of correct answers on both topics rose 18

significantly, with 20% of teachers answering the Lunar Phases question correctly and over 19% answering the Seasons question correctly (Figure 10). This demonstrates the effect of teacher training via experts, though it also demonstrates that more support is needed given that there were still a significant percentage of teachers giving wrong answers to these common science questions.

Figure 10​: The percentage of correct answers given by teachers when asked a multiple choice question on two common astronomy topics, Lunar Phases and the Seasons. The chart shows results collected before the participants attended a Universe in the Classroom CPD workshop, and post­workshop.

Selected Quotes from Teacher Feedback:

“It is a fantastic project. I would recommend it and would love to see similar projects for other areas of science.”

“A great resource. I did not have a STARS workshop but would like to in the future”

“Super resource and training that will be used in other year groups in the Autumn term”

“This is a great resource and it's use has been planned into some year group topics now for the next academic year. It will be used a lot and will support teachers with the teaching of Earth and Space. Thank you for the opportunity to use this resource!”

“We have only recently had this resource but it will be used by other KS2 and Foundation Phase classes in the next academic year. The resources are of a high standard and the training was at a suitable level and very informative.”

“Fantastic resource and I enjoyed the training too. Thanks.”

“Very useful resources and a follow up workshop which pupils enjoyed and learnt a lot from.”

19

“It is very informative, the resources are great and the children benefit a lot from it. Thank you.”

“It was fascinating and the link with Sarah and the students since has been invaluable.”

3.4 Did UitC challenge pupils’ perceptions of STEM?

One of the aims of the project was to address lack of diversity in STEM by tackling perceptions of scientists among young children. Many current programmes which attempt to address the lack of women in STEM (and other minority groups), focus on bringing research academics into secondary classrooms when it may be too late. Furthermore, the pool from which inspirational STEM communicators can be drawn from research academics is extremely narrow, for instance, although 19% of UK physics academics are female, this reduces to only 6% in Wales.

These issues make addressing gender balance and diversity difficult via role models as there are simply not enough scientists to do this. However, at undergraduate and postgraduate level within universities the pool from which we can draw inspirational science communicators is much more diverse ­ including a population with typically 25­35% females, positive male role models, a higher proportion of Welsh speakers, as well as students from a wider range of ethnic groups and backgrounds.

The Stellar Role Model (STARS) ambassador programme was created to able to tackle diversity and gender issues in STEM in Wales on a larger scale than previously possible at the age where impressions and perceptions become fixed. The scheme invited Astronomy, Physics and Mathematics undergraduates and postgraduates from Cardiff University to volunteer as STARS. Figures 11 and 12 demonstrate the diversity of the final group of 34 STARS based on level, STEM subject, gender and ethnicity.

We can see from Figure 11 that 68% of our STARS were undergraduate students whilst 32% were at postgraduate level. Most of our STARS were studying physics rather than astronomy yet were still passionate about using astronomy to inspire children to study STEM.

In Figure 12, the STARS classified themselves as mostly white (91%) with 9% identifying with mixed race and/or black african. By gender, the split was 33:67 for F:M which is significantly larger than the percentage of female academics in physics across the UK (9­25% depending on seniority 11 ). Only 9% of our STARs were Welsh speakers (Figure 13) but these STARS on average did more workshops with schools.

11

​https://www.iop.org/publications/iop/2012/file_53617.pdf

20

Figure 11: ​Left​: Breakdown of the STARS based on their academic level. The STARS scheme invited STEM students from all academic levels: undergraduate (UG), Master’s students (MSc) and PhD students (PhD). ​Right​: breakdown of the STARS based on the subject they are studying at Cardiff University.

Figure 12​: ​Left​: The breakdown of the STARS based on their ethnicity. ​Right​: Breakdown of the STARS based on gender; the percentage is 67% male and 33% female.

Figure 13 Left: ​The breakdown of STARS based on whether they identified as Welsh speakers.

A secondary aim of the STARS workshops was to demonstrate how to implement Universe­in­a­Box investigations for any educators unable to attend a CPD workshop or 21

lacking confidence putting the resources into practise. 58% of teachers claimed to feel more confident using the UitC resources after observing a STARS workshop. The programme also offered benefits to students: each volunteer attended a training workshop that taught them transferable skills in communication, education and presenting.

Evaluation was collected from pupils during these workshops including drawings and personal mind maps (Appendix ​B​). The results of the scientist drawings demonstrated that both male and female children were more likely to draw a male scientist than a female scientist (Figure 14). The majority of the students depicted a scientist as male (97% boys drew male scientist and just over half of girls at 52%), an unsurprising result due to the large gender bias in STEM but perhaps surprising given the early age of the pupils (7­12yrs).

Figure 14 Left: The gender of scientists drawn by each student based on gender. 97% of male students drew a male scientist, whereas just 52% of female students drew a female scientist, demonstrating that even at this age children have been exposed to STEM stereotypes.

However, the number of children abandoning the male stereotype by drawing a female scientist increased noticeably when a female STAR hosted the school workshop. In workshops run by a female STAR the number of both boys and girls drawing a female scientist increased by ~3% compared to a male STAR running the workshop (Figure 15 left).

A similar pattern was seen when the class had a female teacher (Figure 15 right); for boys the number rose by 7% (from 0%), and over 10% more girls drew a female scientist if they were taught by a female teacher.

Age also appears to affect the results, with younger children in years 3 and 4 (98 children) slightly more likely to depict a stereotypical male scientist than older children in years 5 and 6 (153 children) though this is inconclusive given the small differences and the sample sizes of the two age groups (Figure 16).

The drawings taken pre­and­post workshop (Appendix ​B4​) also demonstrated the children's’ perceptions of scientists beyond gender (Figure 17). The four most common features children included on their pre­workshop drawings were, in order of frequency: (i) glasses, (ii) scientific experiment, (iii) lab coat, and (iv) chemistry equipment. The same four features remained the most prevalent in the post­workshop drawings, although the frequency was re­ordered with 22

‘scientific experiment’ most common. The frequency of drawing experiments increased from 27% to 35% whilst glasses decreased from 30 to 16%. This suggests a positive change in children’s perceptions of scientists something superficial (glasses) to seeing them as people who perform experiments and find answers.

Figure 15: ​Above left​: The gender of scientist drawn by each student based on gender of the STAR hosting the workshop. ​Above right​: The gender of scientist drawn by each student based on teacher’s gender.

Figure 16 Left: ​The gender of scientists drawn by each student based on their year group.

Figure 17: The four most common features on primary students’ drawings of scientists create pre (left) and post (right) STARS workshop.

23

3.5: Did UitC provide challenging learning opportunities for primary school pupils?

We can test this using the Personal Meaning Maps (PMM, ​Appendix B5​) that the children completed pre­and post­workshop with STARS. PMM demonstrates the child’s knowledge and perception of space by comparing the frequency of elementary and complex concepts (defined in Appendix ​B6​) described before and after the workshop. As expected, elementary concepts were mentioned more often than complex concepts pre­workshop (elementary 824; complex 260). Post­workshop evaluation saw the number of complex concepts discussed by students more than double, from 260 occurrences to 564 (Figure 18).

Figure 18: ​The number of complex and elementary concepts discussed by children on their PMM evaluation sheets before and after a STARS workshop.

These results clearly demonstrate that although the primary objective of STARS workshops was to inspire children, encourage a passion for STEM and challenge negative perceptions, the workshops were also successful at teaching children challenging concepts and providing them with a more complex vocabulary with which to describe space.

Some of the complex topics introduced in the workshops were too high level for the pupils as seen by the percentage of children who described the concepts incorrectly after the workshop. The most challenging topics to students are shown in Figure 19. These topics were therefore (i) beyond the abilities of the age group (ii) not described at the appropriate level and/or (iii) should only be included in follow up workshops.

24

Figure 19​: The complex topics that were most challenging to students, as determined by percentage of children who described the concepts incorrectly.

Conclusions

Returning to the motivation for the Universe in the Classroom project, we will conclude with a discussion on whether the project achieved the specific targets set in the opening section, What is Universe in the Classroom?

Did UitC provide science continued professional development (CPD), support and advice to schools on science teaching and learning?

Throughout the Universe in the Classroom project 131 welsh primary school teachers attended STEM 8 CPD and 5 twilight training workshops (​Impact Figures​). The teachers were trained in how to use modern, high­quality resources to teach STEM subjects, given advice on how to link STEM topics to other lesson subjects and provided an opportunity to discuss best practise with other educators.

The teachers were also offered ongoing support from the project coordinator in the form of school workshops with student role models and email communication to address any questions or issues.

Did UitC provide support and training for teachers in teaching science?

Universe in the Classroom participant schools were chosen based on defined criteria (​Who Did Universe in the Classroom Reach​) that did not specifically include weak science knowledge. However, the teachers responses to two science questions asked pre­training 25

workshop and again post­training workshop, showed a clear improvement in science subject knowledge in participant teachers (Figure 10).

Did UitC provide challenging learning opportunities and allow for pupils to pursue their own scientific interests?

Universe in the Classroom training workshops notably improved the STEM knowledge and confidence of participant teachers, encouraging them to introduce more science into the

classroom (Figures 5 and 8). The resources provided to schools included over 100 activity guides, many of which were enquiry­based scientific investigations. Additionally, STARS workshops covered a range of complex topics, which were suitable challenging for the target audience, as demonstrated in Figure 19.

Did UtiC Challenge Perceptions of STEM? The Universe in the Classroom project aimed to challenge primary school students’ perceptions of science as a dull, difficult and irrelevant subject and any stereotypical opinions of who can be a scientist, by demonstrating that scientist is an exciting, interesting subject with the new resources and hands­on activities along with workshops hosted by Cardiff University student role models.

Evaluation of student perceptions of science showed a difference in their view of what a scientist looks like after a STARS workshop (Figures 14­16). Their main association with scientists shifted from being something physical (glasses) to being a person actually actively doing scientific research after a STARS workshop.

Did UtiC Improve STEM in Welsh Primary Schools?

Overall Universe in the Classroom made a notable improvement to STEM in welsh primary schools, addressing the demonstrated lack of science resources (Figure 7) by providing 99 schools with new, high­quality hands­on resources, improving the STEM confidence (Figure 8) and knowledge (Figure 10) of 131 teachers, challenging negative student perceptions of scientists (Figures 14­16) and improving relations between scientists and the community through the STARS scheme.

26

Appendix A - Resources used in UitC

A1. Universe-in-a-Box Universe­in­a­Box is an educational toolkit designed by the ​Universe Awareness project to assist educators in bringing sciences and astronomy to 4–10 year old children. The toolkit was developed in response to a demand for practical, interactive and fun resources for bringing science into the classroom. It provides teachers and educators with over 40 practical activities, as well as the materials and models. More information is available at http://www.unawe.org/resources/universebox/​.

A2. Earth Ball and Activity Book

The UNAWE Earth Ball is a simple, innovative resource for use in primary school classrooms, to help students get to know their place on the Earth and in the Universe. It is a light, blow­up version of the planet that comes with an activity book full for teaching astronomy, geography, environmental awareness, global citizenship and even physics and biology. The Earth Ball offer a unique perspective that makes it easy to teach children all about their planet, from its shape to its the axial tilt, its surface composition and more. National borders are not marked, emphasising that we are all

27

citizens of a single planet. The Earth Ball actvitiy book can be viewed at http://www.unawe.org/resources/education/Earthball_booklet/.

A3. Cosmos in your Pocket Cosmos in your Pocket is a small astronomy activity book created by Universe Awareness.

The books are filled with creative activities related to the night sky, including an activity that invites children to design their own constellations, write an ode to their favourite planet or draw a portrait of the Moon. They also include a varieety of fun and education facts about the Solar System. The books are available in 17 different languages, including Welsh. Welsh language copies of Cosmos in your Pocket were distributed as part of Universe in the Classroom. http://www.unawe.org/resources/guides/EU­UNAWE_cosmos_pocket_activity/.

A4. “Where is Spacey’s Planet?”

A story can inspire children, bring out their imagination and make them think. “Where is Spacey’s Planet?” is a storybook about a friendly alien who has landed on Earth's South Pole and cannot find his way home. Along with friends from many different countries, Spacey travels the world in search of his home planet. During this journey, children get a sense of the wonder of the Universe and the world around us, as well as a sense of global citizenship. As the journey continues they pass a big city where the characters cannot see the stars at all, pointing out the importance of preserving the night skies.

A5. Robotic Telescope for Wales Universe in the Classroom and the ​Las Cumbres Observatory worked together to build a robotically controlled 0.4­m telescope for use by Welsh school children. The telescope has been built at the Cerro Tololo Observatory in Chile, a site that boasts some of the best observing conditions in the world and is also home to three 1­m diameter telescopes used for astronomical research by members of the professional astronomical community.

28

The Robotic Telescope for Wales is the first programme of its kind available to primary school students in the UK, and will be a uniting factor for all schools and students involved in Universe in the Classroom, regardless of their social and cultural background, and will complement the other resources to help students explore the Universe.

Above left: The Robotic Telescope for Wales in Cerro Tololo. Above right: A screenshot of the online interface for the robotic telescope for Wales ­ On Sky.

29

Appendix B: Evaluation of UitC Training and Resources A range of evaluation methods were used to assess the implementation and impact of the Universe in the Classroom, including pre­and­post training surveys for teachers, pre­and­post workshop personal meaning maps (PMM) for children and drawings of scientists collected from children.

B1. Pre-teacher training survey At the start of each training workshop evaluation was collected from teachers in the form of a pre­workshop survey (see Figure below). This was designed to collect information about the level of background knowledge in astronomy, the teachers’ experience and confidence in teaching STEM subjects and whether they felt they had the resources to teach hands on activities in science.

B2. Post-teacher training evaluation survey Each teacher was sent a follow­up survey six weeks after they attended a Universe in the Classroom training workshop. The post­workshop evaluation was delayed for six weeks to ensure teachers had time to use the resources in their classroom before providing feedback.

The post­training evaluation was collected using an ​online form which was easy for teachers 30

to complete and return.

The https://uniclass.typeform.com/to/BIdU2F​.

survey

can

be

accessed

at

B3. Evaluation collected from pupils Evaluation was gathered from primary school students at the beginning and end of each STARS workshop to assess the impact on students. Evaluation methods included drawings of scientists to assess the STARS impact on STEM perceptions and stereotypes, and Personal Meaning Maps (PMM) to assess learning.

B4. Draw a Scientist examples:

31

B5. Personal Meaning Maps (PMM) example:

B6. PMM Complex and Elementary concept list Elementary Concepts

Complex Concepts

Earth

Magnetic field

Moon

Jupiter's Great Red Spot

Planets of the Solar System

Satellites

Sun

Lunar features

Stars

Atmosphere

Shooting stars

Space Station

Telescope

Galaxy

Spaceship

Milky Way

Spiked Sun

Black hole

Cloud

Martian ice caps

Rain

Meteorite

Wind

Space Shuttle

NASA

Other moons

Alien

Dwarf planets

Black space

Wormholes

Astronaut

Sunspots

Yellow Sun

Stellar composition

Rocket

Relative size

32

Astronomical ages

Stellar evolution

Solar temperature

Solar flares

Speed of light

Relative orbits within the Solar System

Nuclear Fusion

Types of light

Environment on other worlds

Cosmic dust

White hole

Gravity

Exoplanets

Spaghettification

Eclipses

Cosmic distances

Vacuum of space

Specific missions to space

Earth's axial tilt

Greenhouse effect

Nebulae

The Expanding Universe

Dimensions

Big Bang

33

Universe in the Classroom was made possible thanks to funding from the Welsh Government National Science Academy

The project was run by Sarah Roberts (CU/NSA), Haley Gomez (CU) and Edward Gomez (LCOGT/CU) with the following contributors and partners.

Contributors: Chris North • Charlotte Provot • Stuart Lowe • Gabi North • Peri Jones • Amber Hornsby • Jeni Millard • Matt Allen • Simon Schofield • Olivia Keenan • Ryan Stock • Alex Meredith • Nick Jones • Ellen Hall • Ben McNaughton • Christopher­Rees Jacobs • Zoe Ballard • Chay Davies­Smith • David Lee Jones • Jonathan Sweetland

Partners: ● ​Las Cumbres Observatory (www.lcogt.net): an international astronomy research institute with many years experience providing educational and outreach in science, across the globe. ● ​Universe Awareness ​project (www.unawe.org): an international project based in Netherlands, providing primary school teacher training, and educational resources and activities for 4­10 year olds. Previously funded by European Commission (FP7) this project provides a fully evaluated primary school curriculum and ‘off­the­shelf’ kit for Universe in Box. These kits and the associated curriculum is currently being used in 29 countries. ● ​See Science (www.see­science.co.uk): an educational and enrichment consultancy committed to supporting STEM delivery across the whole of Wales.

Special thanks to Grace Sim and AstroCymru for their help in delivering this project and to Pedro Russo, Hailey Llewellyn and Russell Williams for their continued support and guidance.

34