Skills for Climate Change – Project Evaluation
FINAL PHASE – Learning Tools Review
Dr Jonathan Pratt Steve Matthews Ann Komzolik
20 September 2012
Contact: Dr Jonathan Pratt jonathan.pratt@emergentresearch.co.uk 07725 358933 Emergent Research and Consulting Ltd Canterbury Innovation Centre University Road Canterbury Kent CT2 7FG Registered in England and Wales. Company number: 07452200 Registered address: 31 St George’s Place, Canterbury, Kent, United Kingdom. CT1 1XD.
Executive Summary This evaluation report for the Skills for Climate Change (SFCC) project focuses on a review of the five learning tools developed for the programme by June 2012, namely: 1. 2. 3. 4. 5.
Future-proof Your Career Introduction to the Code for Sustainable Homes Environmental Technology Systems Awareness (Level 3) Solar Renewable Energy Systems Science Waste Management and the Environment
The review is based on learner feedback forms from over 300 learners, interviews with tutors and a panel of peer reviewers with experience of FE, e-learning environments and/or the main topics covered by the learning materials developed. The learner feedback suggests high levels of satisfaction with the learning materials developed in each of the five areas and the learning delivered under the SFCC project. Average feedback scores for the materials overall and in terms of clarity and ease of understanding were consistently 8 out of 10 or above. For three of the tools where more detailed feedback is available, learners report significant improvement in their understanding and awareness of relevant climate change and skills issues following the training. The increase in scores is particularly notable for the Environmental Technology Systems Awareness and Solar Renewable Energy Systems Science tools, suggesting that the desired key learning outcomes for these tools were being achieved. Peer reviewers suggested that the learning tools developed are of a high quality and for the most part are clear and accessible for their intended audiences. They noted however that there is room to improve the interactivity of the materials particularly if they are used in a tutor-led environment. Two of the tools were primarily designed to be used online and this may have contributed to a perceived lack of interactivity when used in a more traditional classroom setting. Most tutors using the materials report that they have worked well and that they have received good informal feedback from learners. Over time, some attempts have been made by tutors to design more interactive sessions around the materials. Learner feedback for the Solar Renewable Energy Systems-Science tool suggested that the level and amount of information covered was proving difficult for a minority of learners. Tutors delivering the learning suggested their delivery has evolved over time to try and take account of these concerns.
Reviewers also suggested that the online tools would benefit from more testing of learner understanding (and particularly their ability to apply what they have learnt) and the use of more video and animations to provide more variety in the nature of the material. Following the interim evaluation report, efforts have been redoubled to pilot the tools more extensively online. However, to date, learner feedback has been limited. In future, it would be worth considering incentives to feedback such as offering respondents a prize draw or High Street tokens.
Contents Page 1.
Context
1
2.
Future-proof Your Career
2
3.
Introduction to the Code for Sustainable Homes
6
4.
Environmental Technology Systems Awareness
9
5.
Solar Renewable Energy Systems Science
12
6.
Waste Management on Site
15
7.
Conclusions
17
Annex 1: Peer Reviewers
18
Annex 2: Peer Review Notes
19
Skills For Climate Change Project Evaluation – Learning Tools Review
1.0 Context This evaluation report for the Skills for Climate Change focuses on a review of the five learning tools developed for the programme by June 2012. The Skills for Climate Change Project has developed new learning materials that focus on previously unmet needs and sought to deliver them in ways (online and mobile) not often used in the construction and building services sector. By June 2012 the project had developed tools in five areas 1. Future-proof Your Career (Level 2) 2. Introduction to the Code for Sustainable Homes (Level 2/3)
3. Environmental Technology Systems Awareness (Level 3) 4. Solar Renewable Energy Systems Science (Level 3/4) 5. Waste Management and the Environment (Level 2) This report updates the interim report reviewing the learning tools. The SFCC project has a clear focus on innovation but such innovation also represents higher risks for the developers (e.g. in terms of effectiveness in delivering learning, likelihood of take-up etc). The first phase of our project evaluation highlighted that feedback on the tools had been quite ad hoc to date and there remained an opportunity to formally review and refine them within the timescale of the project. Emergent therefore sought to review the materials in three ways: 1. Learner feedback – Emergent designed more detailed learner feedback forms for each of the five tools and sent them to the providers. Providers then returned what learner feedback they had managed to collect by the end of June. 2. Tutor feedback – Emergent undertook qualitative telephone interviews with tutors leading sessions using four of the five tools and designed a tutor feedback form for tutors using the Environmental Technology Systems Awareness materials at the National Skills Academy. 3. Peer Review – Emergent identified peers working in similar or related fields and/or in designing e-learning packages (at the Suscon centre in Kent) and arranged for them to review the materials in order to provide constructive feedback (See Annex 1 for details of the Peer Reviewers) In total, some 333 learner feedback forms were returned, although 136 were the generic, simpler forms that had been previously used by the project. Around 80 of the forms were from pilots of the first two tools undertaken in 2011. The table below summarises the responses by tool. Tool 1. Future-proof Your Career 2. Introduction to the Code for Sustainable Homes* 3. Awareness of ETS systems 4. Solar Renewable Energy Systems Science 5. Waste Management and the Environment* *Generic feedback forms
Responses 56* & 30 47 127 41 32
In addition, 7 online feedback forms were received from virtual users of the Future-proof Your Career learning tool. Emergent Research & Consulting
1
Skills For Climate Change Project Evaluation – Learning Tools Review
2.0 Future-proof Your Career The Future-proof Your Career tool was originally developed as an e-learning tool, then adapted to create a set of toolbox talks and flashcards (and more recently as a more traditional classroom based course by one of the delivery partners). It is aimed at all construction & building services engineering workers. Discussions with the provider leading on tutor-led delivery of these materials suggest that most beneficiaries to date have been between the ages of 18-25, are current or recent apprentices and likely to have or be studying for a level 3 qualification. The project evaluation framework outlines the short, medium and long term desired outcomes for the tool. The short-term outcomes summarise what might be called learning outcomes. For Futureproof Your Career, these are principally around improving workforce awareness and understanding of why skills for climate change are needed; what type of skills are needed and how to set about developing the skills. Learner Feedback Some 30 learners filled one of the more detailed feedback forms. On average they gave the materials an overall score of 8.5 out of 10 (see Figure 1). They gave similar scores for the materials in terms of more specific measures such as clarity, realism and usefulness. The scores for interest and relevance were very slightly lower. Figure 1: Learner Ratings for ‘Future-proof Your Career’ Materials
Base: 30 learners
The majority of learners providing feedback suggested that the amount of information given and the level of difficulty of the materials were about right (Figure 2a and 2b overleaf). However, a minority (20%) of learners on this particular tool felt that it may be pitched a little low (too easy).
Emergent Research & Consulting
2
Skills For Climate Change Project Evaluation – Learning Tools Review
Figure 2a: Amount of Information
Figure 2b: Level of difficulty
Base: 30 learners
Feedback from the 56 learners who filled in the more generic original feedback forms was also positive with more than two thirds (68%) rating the training as very good or excellent. Students were most likely to rate the trainers particularly highly (73%). They were most critical of the length of the workshop (16% rated this aspect of the training average or poor). Figure 3 highlights that on a six point scale (from ‘very well’ at point 6 to ‘not at all’ at point 1), learners suggest on average that the course significantly improved learner understanding in each of the three desired learning outcome areas for the tool. Figure 3. Understanding Before and After the Training (Mean score out of 6)
Base: 30 learners Key 1=Do not understand at all 6= Understand Very Well
Verbatim comments from learners were also mostly positive, although several students noted that they would like to have seen more multimedia content, with video and audio clips for example, and interactivity in terms of tasks and activities:
Emergent Research & Consulting
3
Skills For Climate Change Project Evaluation – Learning Tools Review I feel this has helped me a lot. Really enjoyed this course. Today’s presentation was very helpful and understanding. Could make it more interesting, put video clips, something interactive. Bit tedious, get clips and don't patronise us with information that's common sense, give us a task. 97% of the learners providing detailed feedback (i.e. 29 of the 30) said that they would recommend the learning materials to a friend or colleague. By the middle of July, it was reported that there were over 100 registrations for the online ‘Futureproof Your Career’ course, promoted by the Federation of Master Builders. However, only 7 completed feedback forms were received by 19th July. This number is too small to analyse quantitatively. Qualitatively, however we can say that there is no clear evidence that learner scores for the materials online will be significantly lower (or higher) than for tutor-led delivery. Tutor Feedback While the materials have primarily been designed to be used online and at the learners own pace, the learning commissioned to date has been delivered in a tutor-led classroom environment in quite intensive single sessions (usually combining elements of SFCC Tools 1 and 2) with tutors going through the material with the support of a PowerPoint presentation based on the flashcards. Feedback from the tutor was that the materials worked well, the material is pitched at about the right level, and is clear but could be a bit more concise. When the materials talk about competencies, for example, they can be a little repetitive. Some examples may not always be relevant to learners; however it is easy for a tutor working in the sector to augment them with their own examples. Overall, the tutor felt that the materials clearly help learners achieve the learning objectives set. They suggested that the students really seemed to take on board the idea that each and every one of us can contribute to reducing climate change and its impacts The main perceived weakness of the materials was the lack of guidance on or incorporation of more interactive and engaging activities and encouraging them to work in groups for different elements of the learning for example. It should be noted however that the tools were originally designed to be delivered online (with or without tutor support) not in a more traditional tutor-led lecture or classroom style. Peer Review1 The reviewers felt the materials were pitched at the right level for the intended audience (builders, plumbers, electricians etc), were clear and well written, well referenced and included some excellent graphical elements. They noted that the content was quite basic to start off with but built learner’s awareness of the need to adapt their working practices as they progress through the materials. They also suggested that it clearly gives the message that this will require bolt-on courses and continuing professional development rather than complete retraining. The reviewers also felt the materials should be able
1
The Peer Reviewer’s notes for each tool are outlined in more detail in Annex 2.
Emergent Research & Consulting
4
Skills For Climate Change Project Evaluation – Learning Tools Review to impart an understanding of the Government’s commitment to reducing carbon and how this can be achieved in new build and existing homes. The reviewers also felt the idea of working based on flashcards is a good one but that these would be improved if they explained a bit more of the terminology, maybe with small print notes. They noted that some of the materials (e.g. around feed-in-tariffs) will also need updating over time. Using the online tool, navigation was found to be easy between screens and the idea of testing as you move along works well. The reviewers thought the learning checks were relatively easy and would give learners some confidence to move on. The e-learning reviewer also made a number of observations and suggestions that apply more generally across all three online tools (Future-proof Your Career, Introduction to the Code for Sustainable Homes and Environmental Technology Systems Awareness). They suggested: •
The facility to turn off audio commentary (rather than all sound) would be a valuable addition to the learning materials.
•
The inclusion throughout the tools of pertinent video and animations would also be a useful addition in supporting a greater range of learning styles and bringing more of the content ‘to life’.
•
The ability for the user to select their own colour/contrast settings would also make the materials more ‘accessible’ (although with the audio commentary much of the need for a screen reader -although not all- has been removed).
The e-learning reviewer also noted that whilst an iphone app has been developed for one tool, the particular authorware used means the main online content is not accessible in a meaningful way via iPads or iPhones and that when accessed through other mobile devices (e.g. Blackberry & androidvia web browser) it renders poorly on smaller screens. The SCORM2 scoring, they suggested, could also have been used to provide more information on learner progress than a simple pass/fail against each whole tool. The peer reviewers also suggested that as the majority of the intended outcomes for the tools are for the learner to “know” or “be aware” that the learning checks used in the online tools (Multiple Choice Questions) don’t always successfully allow the learner to demonstrate this.
2
Sharable Content Object Reference Model (SCORM) is a collection of standards and specifications for web-based e-learning
Emergent Research & Consulting
5
Skills For Climate Change Project Evaluation – Learning Tools Review
3.0 Introduction to the Code for Sustainable Homes The Introduction to the Code for Sustainable Homes was originally developed as an e-learning tool, then adapted to create a set of toolbox talks, flashcards and a mobile phone application (and more recently as a classroom based course by one of the delivery partners). It is aimed at all construction & building services engineering workers. The evaluation framework suggests that the desired short term (learning) outcomes for the tool were to develop workforce awareness and understanding of what the Code is and how it works, why it is a key driver of the need for workers constructing new homes to adapt and extend their skillset and how understanding the Codes aims and approaches can also be applied to work on existing homes. Learner Feedback As with the first tool, most of the learning undertaken to date for this tool has been in a tutor-led environment rather than online. Discussions with the provider suggest that most beneficiaries to date have been between the ages of 18-25, are current or recent apprentices and likely to have or be studying for a level 3 qualification. Unfortunately, no learners have completed the more detailed feedback forms for this tool, although most of the recent learners who completed the detailed feedback on the Future-proof Your Career learning tool (see Section 2) also completed the Introduction to the Code for Sustainable Homes Training. It is likely that their responses were coloured by both experiences and had they filled in forms for both courses their responses may have been similar to those outlined in Figures 1-3. However, some 47 learners in the previous pilot phase of the project filled out a more generic feedback form. The majority (68%) rated the learning as either very good or excellent (Figure 4). Figure 4: Learner Ratings of ‘Introduction to the Code for Sustainable Homes’ Workshops
Base: 47 learners
Emergent Research & Consulting
6
Skills For Climate Change Project Evaluation – Learning Tools Review
Learners were most likely to rate the trainers particularly highly but were less likely to rate the length of the workshop and the content as highly. Again comments from learners were mostly positive, although several students noted that they would like to have seen more visual content with examples of installations Could do with photographic examples of installation e.g. solar & wind & how cabling would be fed through building materials. Would help if there were a video to show demonstration. It might be better if it had a practical element to the workshop. A few students also felt that it was a lot to take in a short period of time‌ Trainer went through stuff slightly too fast. Venue was too big; found it difficult to hear the trainer. Information given very informative but content too much in short time. Both criticisms are unfortunate, given that the learning materials were originally designed to be online and interactive, and for learners to be able to work at their own pace. Tutor Feedback The learning commissioned to date has been delivered in a tutor-led classroom environment in quite intensive single sessions (usually combining elements of SFCC Tools 1 and 2) with tutors going through the material with the support of a PowerPoint presentation based on the flashcards. This is not what was originally envisaged by the learning tool designers. The tool was designed to be used online and at the learners own pace. Given this context, feedback from the tutor was that the materials worked well, the material is pitched at about the right level (might be more stretching for learners who were not apprentices), and is clear. Some examples may not always be relevant to learners; particularly as most are more involved in retrofits than new homes, however most would see the potential relevance to their work. Overall, the tutor felt that the materials clearly help learners achieve the learning objectives set. Not surprisingly, given the change in delivery method, the main perceived weakness of the materials from the tutors’ perspective was the lack of guidance on or incorporation of more interactive and engaging activities. Peer Review The peer reviewers felt the learning materials were very informative and full of useful information and that the flow was good, gradually increasing knowledge. The information presented would clearly allow the learner to understand the reasons behind the Code, how it is applied and what they could do to ensure buildings meet standards and especially to understand which elements are mandatory. It was thought that even the small business owner concentrating on small build and domestic extensions would understand what can be achieved to raise the energy efficiency of the whole dwelling, how this relates to energy, water, materials etc. and be able to advise his customers accordingly.
Emergent Research & Consulting
7
Skills For Climate Change Project Evaluation – Learning Tools Review On the toolbox talks, the reviewers noted that it is very important to have the key message at the end and suggested they should be applied in every flashback to enhance the summary. They also noted that there was quite a lot of technical terminology and suggested that small print comments on the meaning of technical words might help, as would avoiding abbreviations (such as DER and TER) wherever possible. However, it is perhaps reasonable to expect the toolbox tutor delivering the talk to be able to explain the acronyms and technical terms. The main tool also has a Glossary section. In terms of online interaction, the way you could click onto speech bubbles in the online tool to gain more information and onto hot spots in houses was thought to work well. The reviewers noted that once again navigation was easy between screens. They also suggested that the learning checks were good but could still be improved to show more application of the knowledge gained. The e-learning reviewer noted that presentation and images were of high quality and support the page content well. He noted however that the tool is very large and that the progress bar did not help learners should they progress through the modules in a different order. They suggested that the tool might benefit from being split into distinct units.
Emergent Research & Consulting
8
Skills For Climate Change Project Evaluation – Learning Tools Review
4.0 Environmental Technology Systems Awareness (L3) This learning tool was originally developed in both tutor-led and Virtual Learning Environment (VLE) formats. The learning tool content provides the learning required to achieve the Level 3 QCF unit ‘Understand the Fundamental Principles and Requirements of Environmental Technology Systems’. This unit is used in the City & Guilds 2399-01 Level 3 Award in Environmental Technology Systems Awareness (QCF) and the EAL Level 3 Award in the Fundamental Principles and Requirements of Environmental Technology Systems (QCF). The learning tool has been mainstreamed through the National Skills Academy for Environmental Technologies. The National Skills Academy has also produced an ‘Introduction to Environmental Technology Systems’ learner manual using the SfCC learning tool as the core resource. The project evaluation framework identifies a number of desirable short term (learning) outcomes for the tool, namely that it should improve workforce awareness and understanding of key types of environmental technology system, the fundamental working principles of each technology, the fundamental location/property features required for the potential to install each technology to exist, the fundamental regulatory requirements for each technology and the advantages and disadvantages for each technology. Learner Feedback Discussions with the provider suggested that most learners were current apprentices in Building Services Engineering (electricians, plumbers & air conditioning engineers) and the majority are 1925. The learner feedback forms received for this tool suggested a quite high level of satisfaction overall (with a mean of 9.0 out of 10). Scores for the clarity of the materials were highest; while scores for the apparent relevance of the materials to learners’ current work were noticeably lower (Figure 5). Figure 5: Mean Score (Out of 10) for different aspects of the ETS awareness tools
Emergent Research & Consulting
9
Skills For Climate Change Project Evaluation – Learning Tools Review Base: 127 learners
91% of learners felt the materials were about right in terms of ease of understanding, 7% thought they were too easy. 88% the learners responding felt that amount of information given in the materials was about right (Figures 6a and 6b). Figure 6a: Amount of Information
Figure 6b: Level of difficulty
Base: 127 learners
Encouragingly, learners reported a significant improvement in their levels of understanding in areas corresponding to the desired learning outcomes for the tool (Figure 7). Figure 7. Understanding Before and After the Training (Mean score out of 6)
Base: 127 learners Key 1=Do not understand at all 6= Understand Very Well
Verbatim feedback from learners (on tutor-led courses) was mostly positive, although a number suggested it could be more interactive and/or practical. Emergent Research & Consulting
10
Skills For Climate Change Project Evaluation – Learning Tools Review
More practical learning such as witnessing the process instead of reading about it which is much harder to understand and take in. More interactive. More hands on examples. More on ‘costing’ systems. Tutor Feedback While the materials have primarily been designed to be used online and at the learners own pace, the learning commissioned to date has been delivered in a tutor-led classroom environment in quite intensive sessions over 2 days with tutors going through the material with the support of a Power Point presentation based on the flashcards. A booklet using the same materials has also been prepared for the students to take away. Discussions with one of the tutors revealed that they had incorporated some of their own Q&A sequences into the presentation to make it more interactive. The tutor felt the learning materials were at an appropriate level, were clear and easy to understand and contained about the right level of information. The examples were mostly relevant and realistic, although the diagrams for wind turbines only showed large industrial level turbines rather than the smaller ones students are more likely to meet in a domestic environment. Despite this, the tutor felt the visuals were a key strength of the materials. The tutor suggested that the materials could do with bringing the session together at the end of the day to engage in some kind of activity to use the learning. He noted, however, that the tool is primarily about awareness and will never therefore be that practical (which is why it works well as an introduction to other courses which are). Peer Review The reviewers noted that this tool includes a wide range of technologies and clearly explains how and when they were suitable. The depth of the information was felt to be relevant to a competent workforce and would increase their knowledge and understanding of new technologies whilst noting that further installation training would be required. The reviewers felt the materials will demystify the technologies with language and diagrams that the trained builder/plumber/electrician etc. would understand. They also noted that the learning tool covers a lot of material and the narrative can be repetitive. They suggested replacing some of the text with narration and enlarging the images or introducing animation or videos instead. There was also a suggestion that some more introductory information on the relative costs and benefits, short, medium and long term of different technologies would help learners at this point. The e-learning reviewer noted that the tool again allows progress in a non-linear fashion but that feedback regarding progress (the progress bar) can be inaccurate because of this.
Emergent Research & Consulting
11
Skills For Climate Change Project Evaluation – Learning Tools Review
5.0 Solar Renewable Energy Systems-Science (Level 3) This tool was developed as a tutor-led learning tool (with the Centre for Efficient and Renewable Energy in Buildings at London South Bank University). It is an introductory, non-accredited course that aims to be used with both FE and vocational trainers as well as more advanced practitioners in the industry. The evaluation framework suggests that the short-term desirable (learning) outcomes are the same for both groups; i.e. to develop a more in-depth understanding of the solar resource and the factors that affect it, of solar thermal hot water system science and of solar photovoltaic system science. Learner Feedback Of the 41 learners who provided detailed feedback forms, only three identified themselves as vocational trainers (by filling in a distinct form). In reality the proportion is likely to be higher. Across both groups learners gave the Learning Tool an average score of 8.3 out of 10. Scores for relevance and interest/engagement were slightly lower but still quite high (Figure 8). Figure 8. Mean Score (Out of 10) for different aspects of the Solar Systems Science Tool
Base: 41 learners
Whilst the majority of learners felt that the learning tool materials were pitched at the right level in terms of level of difficulty and amount of information (Figure 9, overleaf), a sizeable minority felt they were too difficult (12%) or included too much information (20%).
Emergent Research & Consulting
12
Skills For Climate Change Project Evaluation – Learning Tools Review Figure 9a: Amount of Information
Figure 9b: Level of difficulty
Base: 41 learners
Once again, it is encouraging to note that learners report significant improvements in their understanding of solar systems science in areas relating to the short term learning objectives for the tool (Figure 10). Indeed the difference in scores is more pronounced than for some of the other SFCC learning tools. Figure 10: Understanding Before and After the Training (Mean score out of 6)
Base: 41 learners Key 1=Do not understand at all 6= Understand Very Well
Comments from learners included: A lot to take in but v enjoyable and a good learning curve.
Emergent Research & Consulting
13
Skills For Climate Change Project Evaluation – Learning Tools Review The equations are quite complicated. Could have done with a little more time/explanation. I found the course very good and knowledgeable and explained very well but I have no previous experience or knowledge of PV and solar panels. A beginners guide at the start would have been a big help. Possibly extend the course for another 1/2 day to reduce the intensity of the training. Tutor Feedback The lead tutors on the course reported that they feel the training had gone very well and they had had good informal feedback from the learners. They acknowledged that there is a lot of information covered with the learners in a very short space of time. They noted that the way the learning has been delivered has evolved over time, partly in response to feedback. The training is completed on a single day to try and minimise disruption to the learners’ working week but the sessions now vary more in style to maintain interest. Learners are shown a range of solar renewable energy systems in operation (on the roof of the CEREB centre) and there is a workshop at the end of the day when learners get to put their learning into practice. Learners are also given more materials to take away with them, including access to live data from the CEREB solar renewable systems and a spreadsheet calculation tool they can use in their working lives. The tutors noted that there is a lot of complex material but that they now focus more on people working at their own pace and leaving some of the more advanced mathematical and scientific context to the supporting materials that people can access if they want to but do not need to go through. They would also like to consider more blended learning delivery in future, providing learners with some materials before they start so that they can orientate themselves prior to the tutor-led delivery. Peer Review The peer reviewers noted that there was a lot of useful information in the learning materials and that the flow was good, gradually increasing the learners knowledge. They noted that the Power Point slides included high quality schematics and visual presentations and that the video sections, whilst of quite a basic quality, offered a good opportunity for learners to consolidate their learning. They suggested that the learning objectives were clearly stated and met in the materials and noted that there were questions posed at the end of the presentation to ensure understanding of the content. However the reviewers suggested there might be too much to complete in one day (assuming that all the materials in the presentation slides are used). They also noted that the level of material seemed to be well above the Level 3 it has been assigned (although nevertheless probably appropriate for the target audience of trainers and advanced practitioners).
Emergent Research & Consulting
14
Skills For Climate Change Project Evaluation – Learning Tools Review
6.0 Waste Management and the Environment This tool was originally developed as a set of toolbox talks in response to a need identified by strategic project partner Higgins Construction PLC, who have since incorporated the materials into their own training programmes internally and with supply chain companies. The evaluation framework identifies a number of short term desirable (learning) outcomes for the tool. Namely that learners are aware of the fundamental requirements of waste management and waste transfer legislation; can understand and apply the ‘waste hierarchy’ and why it should be applied; understand the need to reduce energy use on site and know actions that they can take to reduce energy usage; understand the need to protect the local environment during construction work and know actions that they can take to protect the environment. Learner Feedback The focus for the delivery of this Learning Tool to date has been on workers undertaking an NVQ Level 2 in ‘Removing Non-Hazardous Waste’ in the Olympic Park site (staff fitting and furnishing the Olympic village). It has been used in conjunction with a specific scheme to take on staff who have been unemployed or economically inactive. Beneficiaries are likely to be general labourers ranging from 20s to 40s and many may not have Level 2 qualification. The learners have therefore mostly been non-typical construction workers. Unfortunately, the bulk of the training using this tool took place before the more detailed learner feedback forms were available. However, some 32 learners responded to the generic feedback sheets on the course. Whilst this is a small number, their average scores for the different elements of their workshop experience are very high (Figure 11).
Emergent Research & Consulting
15
Skills For Climate Change Project Evaluation – Learning Tools Review Figure 11: Learner ratings for the ‘Waste Management and the Environment’ Tool
Base: 32 learners
Some 94% rate their overall experience as very good or excellent. No learners reported that any aspect of their training was average or poor. Verbatim feedback from learners included: Constructive and knowledgeable information from trainers explained and answered questions brilliantly. The training was very good and it gave you the knowledge to carry towards future jobs and everyday tasks. I thought that I knew about waste management was basically recycling until this induction. It was interesting to learn how I could be doing a bit more. Learnt a lot. More knowledge about rules on the construction site.
Tutor Feedback Discussion with the current provider reveals that the materials are being delivered using the toolbox talks developed directly by the project (interspersed through their own longer waste management course). The tutor suggested that the materials have been very useful in introducing their own course and what it will cover in more detail. The tutor suggested that some of the content could have been covered in more depth, particularly if it was used with industry savvy labourers rather than completely new recruits. They also suggested Emergent Research & Consulting
16
Skills For Climate Change Project Evaluation – Learning Tools Review that it currently requires an ‘interesting’ trainer – a lack of interactivity meaning that a less confident trainer may struggle. However, a series of suggested Q&As have since been added to the reverse of toolbox talk flashcards. Peer Review The peer reviewer for this tool felt that the toolbox talks were appropriate for their intended audience, including those with little prior knowledge of the subject. They noted that the materials clearly summarise the key issues and provide good introductions to the broader sustainability issues raised. The materials also provide clear, practical information on how to comply with legislation.
Emergent Research & Consulting
17
Skills For Climate Change Project Evaluation – Learning Tools Review
7.0 Conclusions The learner feedback suggests high levels of satisfaction with the learning materials developed for each of the five learning tools and for the learning delivered (by tutors) under the SFCC project. Average feedback scores for the materials overall and in terms of clarity and ease of understanding were consistently 8 out of 10 or above. For three of the tools where more detailed feedback is available, learners report significant improvement in their understanding and awareness of relevant climate change and skills issues following the training. The increase in scores is particularly notable for the Environmental Technology Systems Awareness and Solar Renewable Energy Systems Science tools, suggesting that the desired key learning outcomes for these tools were being achieved. In general the peer reviewers found the online learning materials easy to use with clear and concise instructions. They noted the use of interesting images and graphics. They also liked the idea of Flashcards as a simple pack of cards that helped you remember various facts, although they were not convinced that the cards match with the depth of knowledge in the online courses. The reviewers concluded that the tools were fit for purpose but made a number of suggestions for further improvements. The peer reviewers suggested, for example, that there is room to improve the interactivity of the materials particularly if they are used in a tutor-led environment. Both tutors and learners report that the materials lack a little in terms of interactivity and can be somewhat static and un-engaging (although they are clearly written and include good static visual diagrams and pictures). This is disappointing given the SFCC programmes aim to be innovative and dynamic in terms of engaging small businesses. Two of the tools were primarily designed to be used in an online setting and this may have contributed to a perceived lack of interactivity. Following the interim evaluation report, efforts have been redoubled to pilot the tools in an online setting, although the learner feedback from online users has been limited to date. Nevertheless this raises significant questions for the project going forward. Will the main legacy for the project be the online tools or the more static visual presentation aids? If the former, then the lack of feedback means that a key aspect of the tools remains relatively untested. If the latter, then the materials may need to be refreshed with this in mind, introducing periodic suggestions for group and individual activities and tests.
Emergent Research & Consulting
18
Skills For Climate Change Project Evaluation – Learning Tools Review
Annex 1: Peer Review Team Cristina Blanco-Lion Cristina is a sustainable construction researcher, practitioner and trainer. Most recently she has been working on the accreditation of sustainable construction courses at Suscon, the sustainable construction training centre in Kent. Previously she has worked as a researcher for the Sustainable Urban Design Group and as an Architect for Blanco & Nunez Associates. She has an MA in Architecture, Design and Sustainability from the University of East London. Steve Humphries (e-learning) Steve is highly experienced in both designing and delivering e-learning courses for level 3 and level 4 learners. He has administered VLEs since 2002 and overseen a number of projects customising and implementing institution wide customisations of Moodle (eg Personal Learning Plans), GPS check in block, unified mark book and qualification management. Steve has authored, delivered and gained Edexcel accreditation for a BTEC level 4 "award in e-learning and the VLE. He is also a Hi5 award winner for use of screen capture for learning (JISC and National Excellence Gateway recognition). He is currently Learning Technology Manager for North West Kent College. Sally Clark Sally is the General Manager at Suscon, the Sustainable Construction Training Centre based at the North West Kent College. Sally has 5 years experience of working in housing contracts for a local authority, planning and developing tenant involvement in major refurbishment/retrofit works. She spent 7 years working for a Regeneration Partnership running EU and Government funded research into the skills base and projected skills needs. For the last three years has led the SusCon project which developed, tested and accredited 30 new units of sustainable construction training. More recently, she has been heading up the development of the Higher Apprenticeship in Sustainable Building at SusCon. Noha Nasser Noha is an Associate of Suscon. She is a consultant with a BSc in Architectural Engineering and a PhD in Sustainable Urban Design, Conservation and Regeneration. She has 10 years academic experience including as Project Director of CABE urban design summer school 2007-9. Her specialisms include community engagement, urban design and designing low carbon retrofit schemes for individual dwellings. Charles Nouhan (Waste Management Tool Reviewer) Charles is an Associate of Suscon. He is a former US based real estate broker with a Masters in Environmental Management. He also has 10 years experience as a Recycling Manager at a District Council and was the Chair of the Kent Waste Minimisation and Recycling Forum/Kent Waste Partnership for four. He delivers occasional training to Council customer service reps and refuse collection supervisors and crews.
Emergent Research & Consulting
19
Skills For Climate Change Project Evaluation – Learning Tools Review
Annex 2: Peer Review Notes Tool 1: Future-proof Your Career Criteria Level/Pitch
Relevance of material
Quantity of information/level of detail E-learning review comments in italics
Comments • • • • • • • • • • • •
Clarity/ease of access E-learning review comments in italics
• • •
Quality of presentation E-learning review comments in italics
• • •
Engagement/interactivity E-learning review comments in italics
• • •
Educational assessment
• •
Perceived audience are builders, plumbers, heating engineers, plasterers and electricians – pitch is about right although less text and more visuals/narration may be easier to grasp Pitched at Level 2 but at points seems be more like Level 3 The unit is a bit didactic with insufficient application or testing of knowledge Simple explanation of drivers of climate change and growing professional and consumer demand Basic introduction to areas for climate change intervention that is useful to a beginner or climate-change related occupations looking to skill up Raises awareness of the practical monetary and carbon savings to consumers Clearly suggests ways of seeking greater specialisation in climate change skills Content was basic and built learner’s awareness of the need to adapt working practices Informative and comprehensive content identifying key policies and features. Good referencing of further information and qualifications required. Good use of interactive web links in ‘Resources’ menu but could also be embedded in main body of modules where relevant. Glossary could be expanded to act as an aide-memoire for main content URL’s to further resources embedded into the screen content would assist the learner in connecting to extended reading or possible further research in the subject area. The separate resource section means that the links contained therein are often de-contextualized. The quantity of information and level of detail are good when used on a desktop computer. The amount of text in proportion to the image size is good. The images are relevant and add value to each screen. Clear and straightforward language and font. Easy navigation between screens but not easy to go back to previous screens or start from the beginning once completed Unit Materials are easily accessed through a desktop computer however if using a mobile device the quantity of content and format used e.g. SWF generated by using Articulate have inherent issues with regard to mobile access Excellent graphic of cross-section of home with buttons for more info in Module 2 section on approaches to energy efficiency Some of the more technical images are too small e.g. Module 3 technical skills solar water heating. Presentation is of a high quality. The audio commentary is clear although the facility to turn the commentary off as opposed to removing all sound should be an option. The images are of a good quality. No printer-friendly format for future reference Interactivity is achieved through clicking to visit the next screen/more info and through a drag-and-drop exercise. A bit too much prevalence on giving information rather than encouraging application More activities similar to the drag and drop activity in Module 2 on ‘advice and guidance’ order of approach to energy efficiency methods which creates a more interactive learning experience would enhance the tools Learners will understand that re-training is not necessary but often bolt-on courses will help. Learners will understand government’s commitment to reducing carbon and how this can be achieved in new build and existing homes. Some of the material will need updating; Flashcard 3 for example needs to note that the Feed in tariff was reduced in December 2011. Flashcard 9 could do with revisiting. This is a key diagram and a Pyramid instead of rectangles would improve the emphasis (Reduce energy demand at the bottom of the pyramid would be highlighted as the MAIN action!).
Emergent Research & Consulting
20
Skills For Climate Change Project Evaluation – Learning Tools Review
Tool 2: Introduction to the Code for Sustainable Homes Criteria Level/Pitch
Relevance of material
Quantity of information/level of detail E-learning review comments in italics
Clarity/ease of access E-learning review comments in italics
Quality of presentation E-learning review comments in italics Engagement/interactivity E-learning review comments in italics Educational assessment
Comments • Pitched at a level that even the small businessman concentrating on small build and domestic extensions will understand what can be achieved to raise the energy efficiency of the whole dwelling • The information is more technical but should be accessible to construction industry trades • Pitched at Level 2 but could be increased to Level 3 with greater application and interpretation testing concepts, principles and methods • Very informative information – flow was good gradually increasing knowledge. The information would allow the learner to understand the reasons behind the Code, how it is applied, what they could do to ensure buildings meet standards and especially to understand which elements are mandatory. • Material supports learners to advise customers on energy, water, materials etc. • Range of design solutions to achieve higher code standards made clear in text but could be supported by visuals • Thorough and detailed content explaining at great length the various criteria of the higher Code standards – a little too text heavy could benefit from replacing text with narration and visuals • There is quite a lot of technical terminology: is it possible to add a small print comment on the meaning of that technical word. Is it also possible to avoid abbreviations such as DER and TER (Flashcard 11)? • URL’s to further resources embedded into the screen content would assist the learner in connecting to extended reading. The separate resource section means that the links contained therein are often de-contextualized. • The quantity of information and level of detail are good when used on a desktop computer. The amount of text in proportion to the image size is good. The images are relevant and add value to each screen. • Easy navigation between screens. Good means of learning from mistakes i.e. with corrections identified by reviewing tests • Test is useful but only tests knowledge not application of knowledge (too easy?). • Keeping track of topics and sub-topics got slightly confusing and tedious – suggest option of a summary page for each topic theme ‘in a nutshell’ or subdivisions on the menu bar • This tool is very large. It contains a significant amount of content and would perhaps benefit from being split into a number of tools particularly in light of any scorm data being passed back to the system. This tool also allows progression in a non-linear fashion. Feedback regarding progression therefore is not meaningful to the user should they progress in the order 2, 5, 7 etc… as the progress bar at the base will be inaccurate. • The speech bubbles and hotspots on the houses to gain more information are very successful • Too much text – needs more visual material • Presentation is of a high quality. The audio commentary is clear although the facility to turn the commentary off as opposed to removing all sound should be an option. The images are of a good quality and support the page content well. • Learning checks are easy to use as are hot spots • Simple navigation between screens but limited engagement and interactivity overall • Interactivity is achieved through clicking to visit the next screen/more info and through MC/drag-and-drop quizzes. • A useful introduction to the Code and its various categories and criteria of measurement and intervention • Meets identified learning aims but could benefit from learning checks aimed at applying knowledge
Emergent Research & Consulting
21
Skills For Climate Change Project Evaluation – Learning Tools Review
Tool 3: Environmental Technology Systems Awareness Criteria Level/Pitch
Relevance of material
Quantity of information/level of detail E-learning review comments in italics
Comments • • • • • • • • • •
Clarity/ease of access E-learning review comments in italics
• • •
Quality of presentation
Engagement/interactivity E-learning reviewer comments in italics Educational assessment
• • • • • •
The level is not really addressed at novices/beginners but a more technical audience e.g. micro-combined heat and power systems connection. Key features and concepts should only be included if to be addressed to a general audience. Good example is biomass fuelled system A large amount of knowledge and information but not always easy to digest. Could be supported by better diagrams and animation/video Included a wide range of technologies and clearly explained how and when they were suitable This unit will help de-mystify the technologies with language and diagrams that the trained builder/plumber/electrician etc. would understand. The depth of information is relevant to a competent workforce and would increase their knowledge and understanding of new technologies whilst noting that further installation training would be required. Unlike unit on ‘Future-proof Your Career’ knowledge of costs to install, maintain, long term monetary benefits and savings, as well as carbon savings/efficiency, would be helpful for learners to decide between technologies An overall assessment of how each technology compares to the other would be useful Comprehensive glossary – to be replicated in other units URL’s to further resources embedded into the screen content would assist the learner in connecting to extended reading or possible further research in the subject area. The separate resource section means that the links contained therein are often decontextualized. The quantity of information and level of detail are good when used on a desktop computer. The amount of text in proportion to the image size is good. The images are relevant and add value to each screen. Easy navigation between screens Keeping track of topics and sub-topics got slightly confusing and tedious – suggest option of a summary page for each topic theme ‘in a nutshell’ or subdivisions on the menu bar This tool is of a good size. This tool again allows progression in a non-linear fashion although feedback regarding progression through the tool (progress bar) can be inaccurate because of this. Narration is repetitive – Would suggest replacing text with narration and enlarging image, animation or embedding a video instead Presentation is of a high quality. The audio commentary is clear although the facility to turn the commentary off as opposed to removing all sound should be an option. The images are of a good quality and although generally of a good size there are examples when some images may be too small. Some interactivity is achieved through clicking to visit the next screen/more info and through MC/drag-and-drop quizzes. Information often presented in a static way – overall quite limited interactivity in this unit Meets the intended learning objectives but unit remains quite didactic, particularly if used as slides in a classroom setting rather than as an online tool (as was originally intended).
Emergent Research & Consulting
22
Skills For Climate Change Project Evaluation – Learning Tools Review
Tool 4: Solar Renewable Energy Systems Science Criteria Level/Pitch
Comments • Pitched at undergraduate level. • Well above level 3 delivery. • Technical information based on physics of solar technology. • Videos were rather basic (at lower level than information on slides)
Relevance of material
• • •
Quantity of information/level of detail
• •
Very informative information – flow was good gradually increasing knowledge Material support learners to understand how solar renewable technologies work. Materials support learners to understand the limitations of the solar technologies and relate to orientation, tilt, seasonal changes and occupant behaviour. Very thorough and detailed explanation If all modules covered in one day - too much detail to absorb.
Clarity/ease of access
• •
Powerpoint Videos
Quality of presentation
• • • • • • • • •
Excellent quality powerpoint presentation Clear schematics – excellent visual presentations Videos were basic (lower level than presentation) but provided an opportunity to consolidate learning. Learners without prior overview training would struggle to remain engaged. Although Powerpoint by itself does not allow very much interactivity – a good presenter can ensure learners are engaged. Videos – good use of visuals A good in depth look at the science behind solar renewable technologies. Learning objectives clearly stated and met within the presentation. Assessment tools not described but questions posed at the end of the presentation to ensure understanding of content.
Engagement/interactivity
Educational assessment
Emergent Research & Consulting
23
Skills For Climate Change Project Evaluation – Learning Tools Review
Tool 5: Waste Management on-site (Toolbox Talks only) Criteria Level/Pitch Relevance of material
Comments
• •
Is appropriate for those with little prior knowledge of the subject. The ‘What’ sections clearly summarise the core issues. Other sections offer good introductions to the broader sustainability issues raised by the subjects in each toolbox talk. Where appropriate, the do and don’t sections provide practical information on how to comply with the legislation Level of detail in each toolbox talk is probably about right for the audience.
•
Tool box talk structure is good/very good in terms of course introduction, topic layout and course summary.
Quality of presentation
•
Good
Engagement/interactivity
•
NA
Educational assessment
•
The toolbox talks fulfil the needs identified and would be fine to use in their current form – but some amendments are recommended if it is practical to do so:
Quantity of information/level of detail Clarity/ease of access
• •
Toolbox Talk 25: Should point out that SIC codes relate to industries not occupations. The Why? section could include a statement such as ‘The recovery of materials for reuse or recycling ensures that an adequate supply of essential materials will be available in the future as they become scarce’. Post delivery checklist Q 3 could include reduced site vehicle traffic as it relates to both site safety and impact on those neighbouring the site.. Toolbox Talk 27: In the Why? Section, strictly speaking CO2 already in the atmosphere is driving current global warming, CO2 emissions could accelerate the rise in global temperature averages in future. Further section, note that cement production is the source of 5% of man-made CO2 emissions not global CO2 emissions Toolbox Talk 28: See first comment on Why section in Talk 27. Further section, there could be a mention of how masonry rubble such as brick, block and cement are recycled to produce similar materials that have a significant recycled content with carbon-saving and other benefits. Toolbox Talk 29: See first comment on Why section in Talk 27. There is no mention of the sustainable use of water in plastering/dry-lining operations. Sustainable use of water reduces multiple environmental impacts (e.g. excess water use, pollution) and reduces the costs and time associated with over-use and clean-up operations. Toolbox Talk 30: See first comment on Why section in Talk 27. Toolbox Talk 31 & 32: See first comment on Why section in Talk 27. There is no mention of the larger concept of conservation. Although not necessary for the module an overall understanding of the concept by practitioners will result in across-the-board environmental savings, particularly in the area of energy use. Toolbox Talk 33: See first comment on Why section in Talk 27. The Why section also makes specific reference to Higgins Construction PLC. The first bullet point should be modified for more general use.
Emergent Research & Consulting
24