BML323: Designing, Developing and Delivering Training Materials
Monitoring, Evalua:on and Feedback
Dr Andy Clegg
Aims To iden(fy the purpose and ra(onale for evalua(on and training To cri(cally evaluate the key elements of the monitoring, evalua(on
and feedback process
To consider methodological approaches against established
frameworks for monitoring and evalua(on, and their applica(on to different kinds of training events/scenarios
To consider the prac(cal reali(es of the monitoring and evalua(on
process, and devise appropriate monitoring and evalua(on strategies for suggested training scenarios, with reference to case studies
Setting the context…
Monitoring and Evaluation Ac:vity 1: • Iden(fy the context(s) for monitoring and evalua(on…
Learning Outcomes Setting the Context for Evaluation
—  Learning outcomes are goals that describe how a trainee
will be different because of a learning experience. More specifically, learning outcomes are the knowledge, skills, attitudes, and habits of mind that trainees take with them from a learning experience
Learning Outcomes Setting the Context for Evaluation
Learning outcomes are goals that describe how a trainee
will be different because of a learning experience. More specifically, learning outcomes are the knowledge, skills, attitudes, and habits of mind that trainees take with them from a learning experience Relate to the business case for training/ what has
necessitated a training intervention?
Learning Outcomes Setting the Context for Evaluation Specific
outcome is focused on a specific category of trainee learning/development
Measureable
data (quantitative/qualitative) can be collected to measure trainee learning
Attainable
the outcome is attainable given the training experience
Results-focused
learning outcomes are assessed against an appropriate assessment framework/marking schematic/ROI framework
Tailored
learning outcome reflects the wider programme aims/levelness [Adapted from the University Oregon, 2015]
Formative v Summative Formative The goal of forma(ve assessment/evalua(on is to monitor
the progression of trainee learning to provide on-going feedback that can be used by instructors to improve their teaching and by trainees to improve their learning Summative The goal of summa(ve assessment is to evaluate student
learning at the end of an instruc(onal unit by comparing it against some standard or benchmark
Context
Internal • Trainees are employees of a specific business
External • Trainees are separate businesses
Context
The purpose of evaluation…
Monitoring and Evaluation Ac:vity 2: • What is the purpose of evalua(on? • What elements of training are you actually monitoring and evalua(ng, and why? • What the key ques(ons to ask for effec(ve training and learning evalua(on?
The Purpose of Evaluation Key Themes To determine whether the training and development objectives are
being met To determine the effectiveness of the different components of
training and development programme (e.g. contents, training aids, facilities and environment, programme schedule, presentation style, the instructor etc) To determine whether the training and development programme
justifies the cost To decide who (number and type of potential participants) should
participate in future programme To assess which participants gained the most or the least from
specific programmes
[Topno, 2012]
The Purpose of Evaluation Key Themes To gain practical insight in order to design, develop and deliver
more effective future programmes To conform to policy guidelines and documentation of training and
development efforts To check the extent of transfer of learning i.e. the extent to which
a trainees applies to his/her job To determine if the training programme maps to the needs the
trainees/organisation
[Topno, 2012]
The Purpose of Evaluation Key Themes To validate tools and methods of training needs analysis To confirm/revise options available To confirm or revise training strategies chosen To determine trainee and trainer reactions To assess the acquisition of knowledge, skills and attitudes To assess trainee's performance in the workplace To determine whether organisational goals are being met
[Simmonds, 2003]
Effective Monitoring and Evaluation Assessing the Learning To what extent were the iden(fied training needs objec(ves
achieved by the programme? To what extent were the learners' objec(ves achieved? What specifically did the learners learn? Evidence of aJainment of learning outcomes? Evidence of implementa(on of learning outcomes in work/
business context?
Effective Monitoring and Evaluation Organisational Context How successful were the trainees in implemen(ng their
ac(on plans? To what extent were they supported in this by their line
managers? To what extent has it supported professional development? To what extent has the ac(on listed above achieved a Return
on Investment (ROI) for the organiza(on?
Frameworks for Evaluation Kirkpatrick’s 4 Levels of Evaluation
ReacFon of Trainee
Learning
Behaviour
Results
Kirkpatrick’s 4 Levels of Evaluation Level 1: Reac:on of Trainee a measure of sa(sfac(on
Guidelines
• Determine what you want to find out • Design a form that will quanFfy reacFons • Encourage wriNen comments and suggesFons • Get a 100 percent immediate response • Get honest responses • Develop acceptable standards • Measure reacFons against standards and take the appropriate acFon • Communicate reacFons as appropriate.
Kirkpatrick’s 4 Levels of Evaluation Level 2: Learning
learning as the extent to which par(cipants change aOtudes, increase knowledge, and/or increase skill as a result of aJending a program
Guidelines
• Use a control group if it is pracFcal • Evaluate knowledge, skills, and/or aVtudes both before and aWer the program • Use tesFng to measure knowledge and aVtudes, and use a performance test to measure skills • Get a 100 percent response • Use the results of the evaluaFon to take appropriate acFon
Kirkpatrick’s 4 Levels of Evaluation Level 3: Behaviour
the extent to which a change in behavior has occurred because someone aJended a training program
Guidelines
• Use a control group if that is pracFcal • Allow Fme for a change in behavior to take place • Evaluate both before and aWer the program if that is pracFcal • Survey and/or interview one or more of the following: trainees, their immediate supervisors, their subordinates and others who oWen observe their behavior. • Get a 100 percent response • Repeat the evaluaFon at appropriate Fmes • Consider cost versus benefits
Kirkpatrick’s 4 Levels of Evaluation Level 4: Results
the final results that occurred because a person aJended a training session
Guidelines
• Use a control group if it is pracFcal • Allow Fme for results to be achieved • Measure both before and aWer the program if it is pracFcal • Repeat the measurement at appropriate Fmes • Consider cost versus benefit • Be saFsfied with evidence if proof is not possible
Methodologies‌
Monitoring and Evaluation Ac:vity 3: •  Iden(fy qualita(ve and quan(ta(ve methodologies that could be used to support the monitoring and evalua(on process
Effective Monitoring and Evaluation Indicative Approaches Personal inventories and learning logs Portfolio planning Diagnostic testing Observations of practice Group discussion/focus groups and analysis Interviews Mentoring programmes
Evaluation Scenarios Scenario 1:
Scenario 4:
• Measure the impact of the Learning and Teaching Conference/Fes(val in June
Measure the impact of an
Scenario 2: • Measure the impact of a sector training event focusing on running a green business Scenario 3: • Measure the impact of a staff training event on using Moodle
organisa(onal training programme focusing on managing the visitor experience Scenario 5: Measure the impact of a sector
training event focusing on maximising social media Scenario 6: Measure the impact of a sector
training event on how to run a sustainable event compliant with ISO 2012-1
Effective Monitoring and Evaluation Key Themes Honest, transparent, criterion-based Trainers must also be open to feedback and encouraged to reflect
on their own performance Holistic approach – not a simple straight line; multi-dimensional
and dynamic Creativity in choice of approach to capture feedback – 'fit for
purpose' Embed into a culture of 'continuous improvement'
Case studies‌
Case Study 1:
Employability Journeys in Dance —  Evalua(on centred around
Dance Alumni looking at aOtudes towards employability skills
Case Study 2:
Rural Welcome —  Evalua(on of a skills and
training programme for rural tourism businesses
Case Study 2: Rural Welcome
Case Study 2: Rural Welcome
Case Study 2: Rural Welcome
Case Study 2: Rural Welcome
Case Study 2: Rural Welcome
Case Study 3: NSS Ac:vity 4: Q6: Assessment arrangements and marking have been fair • What does this mean to you? • What criteria elements are you judging on? • What does fair mean? • What could be done to improve scores in this area?
Case Study 3: NSS Ac:vity 5: • What makes effec(ve feedback? • What type of feedback works best for you?
Evaluation Scenario Ac:vity 6: • You have been asked to develop a new model by which the Business School can effec(vely assess teamwork, so that individual contribu(ons can be iden(fied and assessed fairly • What criteria would underpin your model and how would you implement it?
Evaluation Scenario Ac:vity 7: •  What evalua(on are you planning for your own training event?
Summary By the end of this session you should be able to: § Iden(fy the purpose and ra(onale for evalua(on and training § Cri(cally evaluate the key elements of the monitoring, evalua(on
and feedback process
§ Consider methodological approaches against established
frameworks for monitoring and evalua(on, and their applica(on to different kinds of training events/scenarios
§ Address the prac(cal reali(es of the monitoring and evalua(on
process, and devise appropriate monitoring and evalua(on strategies for suggested training scenarios, with reference to case studies