Getting your clients ready: Defining success Reading Summary
Course: Design and Evaluation of Training Workshops and Programs Prof. M. Ed. Verónica Castro R. October, 2013
Student: Xinia Nagygellér Jiménez
Chapter 3: Investigating needs and planning a workshop It is essential to provide direction and purpose to workshops and training programs in order to achieve the expected success. Klatt (1999) identifies 9 elements to be considered for this purpose in the planning of such programs: Workshop and learning needs: to deal with problems and seize opportunities Agenda, purpose, outcomes: they specify the plan for the activity, the aim or intention for it, and what it is expected to achieve. Content, process: the first refers to the material to be provided to participants and the latter to how they will work together. Design: the methods, activities or exercises, strategies and processes to be employed Capability: the set of abilities, skills and attitude in all the people that have a part in the activity Feedback: the reaction and information (either evaluative, descriptive, general, specific, immediate or delayed) that participants provide regarding an action or activity. All these elements fit together and should be considered in sequence as a general guideline when planning a workshop or training program.
Investigating Workshop and Learning Needs It is necessary to determine the needs of an organization before deciding on the content and the processes to follow in a workshop or training program. In this way, what is to be achieved and how to know that it has been achieved become clear and help determine necessary changes in the workplace. A) A list critical questions that aid in needs analysis would aim at: 1. Defining the problem or opportunity and the organizational changes needed 2. Foreseeing the value and likelihood of achieving these changes 3. Identifying how much of these needed changes involve employee’s willingness to do the job, opportunity to do the job, and ability to do the job 4. Determining which behavior changes will most support the desired organizational changes 5. Identifying the factors that influence or block the wanted behaviors 6. Defining what participants need to know or the skills and insights they require to change behaviors 7. Deciding on who needs to take part of the workshop and who influences the needed changes the most 8. Planning the process and materials needed to provide the knowledge, skills and insights wanted
B) The «Felt Needs Trap» It order not to miss the focus of a workshop, a leader should not base it solely on the needs participants feel they have, as they may not be able to see their «blind spots» or where their incompetence lies. It is especially those people who are unconsciously incompetent that need workshops, and the workshop leader should make sure they are learning-ready. C) A Model for Understanding Learning Needs
Unconscious competence Competence is a good habit
Conscious competence Awareness of steps to perform
Conscious incompetence Learning needs are recognized
Unconscious Incompetence Incompetence is a bad habit
D) Distinct Approaches for Investigating Workshop and Learning Needs
The Objectivist Approach Based on objective reality in the environment Learning needs are expressed in behavioral terms and they are stable
Similar to single-loop learning It’s explicit, based on established goals
The Interpretative Approach Considers changing, complex variables Sees alternative choices for action Requires conceptual and perspective abilities
Similar to double-loop learning It’s implicit, based on reassessing goals and finding new ways to deal with the situation
Steps in both approaches 1. Gathering data 2. Analyzing the data 3. Writing workshop outcomes 4. Designing and conducting the workshop to meet outcomes 5. Support and reinforcement of participants’ new behaviors on the job by supervisors 6. Evaluating the workshop
E) A technique for probing workshop and learning needs This implies asking why 5 times in order to investigate workshop and learning needs. It is often an annoying technique.
Planning workshops and training programs
a. The process cycle: it is a way to plan a workshop or training program from its very beginning till the end. The cycle includes: b. Purpose: establishing what the focus or intention of the workshop or training program is and what it is not c. Outcomes: the results of the workshop and leadership in measurable and observable ways. They must relate to the purposes established and be clear. d. Steps before, during and after the session(s) have to be planned to reach the expected outcomes. These steps keep the activity on track. e. Capabilities: defining who is to attend the activity, what attitudes and behaviors they should exhibit, as well as the necessary resources f. Feedback: from stakeholders and evaluation. It should be foreseen, and how the workshop will be evaluated and how that evaluation will be used must also be planned.
b) Gathering information in a group-putting the process cycle to work Brainstorming is a good first step to begin the process cycle. It can be used to define what the purpose of the workshop is. Based on this purpose, the expected outcomes and then the necessary steps to reach them should be thought out. Next, the capabilities and resources that are needed for the workshop are defined. And at last, decisions in regards to what evaluation is to be used and what feedback is expected from it are made. Following this cycle, the design, the agenda, and communication about the workshop or training program can be better directed and prepared.
Chapter 4: Planning to evaluate a workshop Klatt (1999) claims that evaluation is a necessary step to achieve success in a workshop and to find out what needs to be improved as a workshop leader through the feedback that it can provide. What has to be evaluated? Klatt (1999) mentions two main aspects: 1. The value of the workshop or training program to the participants and to the organization. This means, evaluating if the expected outcomes have been reached. 2. The performance of the workshop or training-program leader. The ideal would be to focus on achieving a couple of outcomes for the leader’s development and evaluate if they have been reached. The author goes further to explain four different levels of evaluation. These levels have been proposed in a hierarchical way, and each one implies a higher degree of complexity than the previous one(s).
Four Levels of Evaluation
Level 1 Reaction
Level 2 Learning
Level 3 Behavior
Level 4 Results
• This level “measures feelings such as attitude, energy, enthusiasm, interest, and support” (Klatt, 1999, p.127) the participants of the program have experienced (their reactions), which indicate how well they have accepted the process and what has been presented to them. • The workshop leader’s evaluation at this point will also serve to evaluate his effectiveness as a leader. • Data can be collected through end of the workshop evaluation forms, tailored to the context and completed in the last 5-10 minutes of the workshop. They can include items to collect both quantitative and qualitative data, and participants may be asked to provide their names or leave it optional.
• Level two focuses on awareness and understanding. It deals with measuring whether or not participants have acquired skills or have learned what they have been taught. This can be done through simulations in the case of skills, or with written tests in the case of principles and facts. • Tests of this type can be applied before and after the workshop is presented in order to compare them statistically and look for improvement in their mean scores. • At this point, the workshop leader can also evaluate himself by reflecting on what he has learned in the process and on what can be changed as a result.
• Level 3 evaluation pays attention to those changes in behavior on the job that can be attributed to an individual’s participation in a workshop or training program. • This evaluation can be done by observing participants’ performance on the job, by having them demonstrate skills in a workshop or at work and by having a supervisor complete performance assessment forms that include ample description of behavior pre or post workshop or training program. • By assessing behavior, the workshop leader can also assess to what degree he has reached his leadership outcomes and objectives.
• The focus of this level is on the benefits of the workshop for the organization. • It involves examining the results the workshop or training program has promoted in the organization by having participants take advantage of the program, learn its content and change behaviors on the job. However, it is a difficult task to measure results, for there are many variables that cannot be controlled and which may influence the results obtained or which may make it hard to define if the results observed are really a product of the individuals’ participation in the workshop or training program. • As a result, it is possible to assess changes in tasks and skills (observable behaviors) at this level, and because of this fact, not all workshops will be evaluated either at level 3 or 4. When these levels of evaluation are conducted, however, there should be a well defined purpose and clarity on what is to be measured. Also, time for behavior changes has to be considered before evaluation at these two last levels. • Skills in surveying, research, statistics, diagnosis and interviewing are necessary in this kind of assessment. • Making use of measurements and control groups before and after the workshop is recommended, and paying attention to those sources of information in the work environment is also needed. • Not only participants but also supervisors, workmates and others can be surveyed about changes on the job and in the organizational results in order to assess the reach of the training or workshop provided.
Evaluation in Perspective The following are some aspects of evaluation that Klatt (1999) considers important points to pay attention to. a. The “Inside Stuff” on Evaluation: The use of forms and other written evaluations provides only basic information about the success of a workshop or training program. But talking to participants and assessing their performance on the job and the results of their behavior changes at levels 3 and 4 provide a better idea of improvements at the organizational level. Nevertheless, assessing at those levels is expensive and difficult; it requires a good deal of time and resources. Even when the benefits a workshop will bring to an organization are clear and obvious, some kind of formal evaluation must be performed. b. The Leader’s Role in Evaluation: Judgment and description of the leader’s role are the result of evaluation, and problems and anxiety may emerge from this. At level 1-reaction, the level of trust the group has developed with the leader and how he asks for evaluations may influence the results. Also, the way he feels about he group and about himself will affect them. Regarding this, the author advises to pay attention to those who will be honest and direct and who can be trusted at the moment of interpreting workshop and training program evaluations.
c. Client Bashing-Not the way to go Here, Klatt (1999) emphasizes the idea that responsibility for the success or failure of a workshop is a shared point by workshop leaders and their clients. Hence, he reminds his readers of the importance of not blaming others for failure, but sharing responsibility and taking the experience as a way to learn and grow. d. Attribution Theory This theory focuses on how developing trust to a high degree and establishing rapport with participants will pay off by gaining their support, appreciation, and even forgiveness when things do not turn out as expected. As long as the workshop leader is liked by participants, they will evaluate his performance and what they have acquired in the program in a more positive way. e. Evaluating the Performance of Participants The use of grading to evaluate participants’ performance is advised. However, grades may become more important than learning when participants center their attention on them. Also, the leader will not be considered their colleague or coach if participants feel they are been judged. And the environment in the workshop or training program can also be affected if participants become competitive instead of collaborative and cooperative with others. So, instead of enjoying the experience and process, they may feel under pressure to obtain a good grade.
Chapter 5: Contracting outcomes and marketing prior to start-up In this chapter, Klatt (1999) claims that it is necessary to make decisions and agree on outcomes and the roles of the different stakeholders in a workshop or training program in the form of contracts for it to be sucessful. Contracts should be done with: a) Sponsor, administrative and end-user clients
Sponsor: person or group that approves and sponsors the workshop/training program. Administrative client: the one in charge of arranging and coordinating the logistics of the workshop. End-user clients: the participants in the workshop or training program.
b) Prisoners and vacationers
Prisoners: participants included by their bosses, with no interest in the program Vacationers: participants who sign up for the workshop to have time off from work
c) Yourself—the workshop leader
A commitment to learn from the experience and improve at least two aspects in leading a workshop (facilitation, preparation, leading role)
How to contract with different clients It is important to agree on some points related to what clients want and what the workshop leader can offer them. Klatt (1999) presents the ways to contract with the three kinds of clients mentioned before: a) Sponsor and administrative clients: A short meeting and a short memo to clarify understanding, expectations and agreement are appropriate. The memo should be clear, and it should «confirm the ‘partnership’ role of [the] sponsor client in the success of a workshop, along with the responsibilities for action and follow through» (p.156). A confirming letter used for this purpose would include confirmation of aspects such as date, time, location, and name of the workshop leader, as well as a request for the resources the organization should supply and what the workshop organizer will supply. It may also include the cost of the program and diagrams for sitting arrangements in the location of the workshop. It would end by offering further information and clarification through contact numbers.
b) Participants or end-user clients: ď ś Through a pre-meeting with participants, they can get information on the workshop and prepare for it. It serves to meet participants, define the purpose and outcomes of the workshop or training program, discuss its benefits, and encourage participants to sign up. Also, it is an opportunity to develop and distribute pre-workshop assignments, questionnaires, materials, agenda, and workbook. Expectations and needs can also be discussed at this point and administrative aspects dealing with the agenda and location can be clarified. This meeting may besides serve to set a positive environment and establish credibility as a presenter and rapport with these clients. ď ś Pre-workshop questionnaires would serve to collect information on what participants want to obtain from the program. These should be analized and discussed when the workshop starts. The most important step with this group is to make them realize the benefits the workshop has for them. ď ś A confirming letter could be useful for this group too, and it should contain information about the program logistics, what it has been designed for, what they need to do to prepare for it, and contact information for clarification of doubts.
A Pre-workshop participant questionnaire would be useful to obtain information on the «participants’ backgrounds, relevant experience, and specific training interests and concerns» (Klatt, 1999, p.166). This tool would include the purpose of the workshop and of the questionnaire itself, instructions on how to fill it out and contact information for doubts or clarification. Moreover, «one pagers», which are short, one page marketing circulars or program overviews, can be used to stress the benefits to participants, specify who the workshop is intended for, what participants will learn from the experience, and the logistics involved. Finally, in order to make a good impression and show professionalism as a workshop leader, Klatt (1999) suggests watching the quality of the materials used for marketing. They should not only be high quality, but also they should be written following the rules of grammar and spelling and use the active voice.
References Klatt, B. (1999). The Ultimate Training Workshop Handbook. New York: McGraw Hill.