https://www.aretezoe.com/
TRAINING SOLUTIONS Pharmaceuticals | Healthcare
PART IV – IMPLEMENTATION PART V – EVALUATION
07/FEB/2018
1
https://www.aretezoe.com/
Drug development and commercialization processes are complex and require advanced working knowledge in multiple domains. Teams need to be trained in a manner that enables optimal function that meets the needs of the enterprise, including operational efficiencies, continuous improvement, regulatory compliance and the need to keep up with innovative approaches and new coming technologies.
07/FEB/2018
2
https://www.aretezoe.com/
Systems Approach to Training Evaluate
Flexible, outcome-oriented system Instructional system shall determine needs and priorities, develop effective and effective solutions to achieving these needs, implement them competently, and asses how well the outcomes meet the specified needs.
Implement
Develop Design
Analyze 07/FEB/2018
3
https://www.aretezoe.com/
Determine the effectiveness and efficiency of the training program
EVALUATE Identify critical issues & topics
•
All phases subject to evaluation
•
What is the program’s impact?
•
Any instructional deficiencies?
•
Multiple tracks
•
Any major changes that would affect the program?
•
Document review
•
Forms designed to answer specific concerns
•
Student feedback
•
Student performance
•
Instructor feedback
•
Will evaluation affect any important decisions?
Select evaluation approach •
Objectives-oriented
•
Instructor performance
•
Management-oriented
•
Post-graduate survey
•
Operational test & evaluation
•
After instruction report
•
Collect, organize, collate data
•
Quantify and interpret data
•
Statistical evaluation
•
Summarize data for review board
•
Identify strengths and problem areas
•
Determine and record solutions
•
Course history •
Enrollment rosters, student data sheets
•
Tests, after-instruction reports
•
End of course critique summary
•
Graduation roster
Analyze, interpret & manage data Plan
Conduct evaluation
Course content review Evaluation summary
Course revision plan 07/FEB/2018
4
EVALUATE
Document Review
Analyze Phase Evaluation
Design Phase Evaluation
Develop Phase Evaluation
Implement Phase Evaluation
Analyze & Interpret Data Manage Evaluation Data Course Content Review 2/2/2018
https://www.aretezoe.com/
5
https://www.aretezoe.com/
EVALUATE
Individual Training Standard
Training Manual
Document Review
Course descriptive data
Program of Instruction
Analyze Phase Evaluation
Design Phase Evaluation
Develop Phase Evaluation
Implement Phase Evaluation
Master Lesson File Analyze & Interpret Data
SOPs and Evaluation Plans Manage Evaluation Data
Inspection/audit reports Course Content Review
Record of proceedings 07/FEB/2018
6
https://www.aretezoe.com/
Front-End Analysis to collect, collate and report job performance data
ANALYZE New laws and standards
Task is a job behavior
Set of training standards
New policies, requirements
Logical and necessary unit
Subject Matter Expert Conference
New technology, systems
Observable & measurable
Participation of resident experts is critical
New projects, new people
Specific result
Determine list of tasks for instruction
Mergers & Acquisitions
Define the task
Conditions, standards & performance steps
Business expansion
Prerequisite skills, ability
Organizational roles & responsibilities
Organizational changes
Conditions and standards
Determine instructional setting
Evaluations, audit
Performance steps
Resource requirements
Job Analysis
Task Analysis
Requirements & Responsibilities
What are the job requirements?
What are the tasks required to perform the job?
What are the R&Rs of each element in the training establishment?
Training standards
07/FEB/2018
Task inventory
Individual training standards
Target population description
Corporate Training Manual
Performance criteria
Train the trainer program 7
https://www.aretezoe.com/
OUTPUT
Corporate Training Manual Target List
population description
of tasks for instruction
Individual
Training Standards
Conditions
& Performance steps
Instructional Roles
07/FEB/2018
setting
& Responsibilities, staffing
8
https://www.aretezoe.com/
EVALUATE
Does the training standard reflect the job task?
Document Review
Are the prerequisite knowledge and skills accurately represented?
Does the training environment accurately represent the job?
Does the target population accurately represent those who perform the task? 07/FEB/2018
Analyze Phase Evaluation
Design Phase Evaluation
Develop Phase Evaluation
Implement Phase Evaluation
Analyze & Interpret Data Manage Evaluation Data Course Content Review
9
https://www.aretezoe.com/
Corporate Training Manual becomes the input to the Design phase.
DESIGN Knowledge, skills, attitudes
Published materials
Sequence terminal learning objectives (TLO)
Target audience
Subject matter experts
Enabling learning objectives (ELO)
Methods of instruction
Learning analysis worksheet
Logical transitions from one skill to the next
Methods of evaluation
Individual training standard
Grouping based on shared elements
Gather data
Sustainment period
Sequencing: hierarchical, logical
Review students’ background
Standard & Conditions
Sequenced TLOs serve as a course structure
Summarize entry requirements
Test items
Target Audience
Learning Analysis
Who is coming for instruction? What are their Knowledge, Skills and Attitudes?
What will be taught, evaluated and how?
Sequence learning objectives What order shall the instruction follow to maximize resources and transfer of learning?
Learning objectives Test items
Target population description 07/FEB/2018
Methods & Media
Sequenced Terminal Learning Objectives 10
https://www.aretezoe.com/
Components of a learning objective BEHAVIOR
CONDITION
STANDARD
1 action verb & 1 object
Aiding: reference materials, tools, equipment, job aids
Describes level of proficiency the student has to achieve
Free of ambiguity Observable & measurable Bloom’s Taxonomy verbs Simple, clear & concise Realistic performance in the instructional environment
07/FEB/2018
Limiting: specific resource or information not available Environmental:
Location & facilities Social context – teams
Implied: LO as described implies the existence of equipment and resources needed to operate it
Completeness: quantitative statement that indicates acceptable standard Accuracy: use references that describe acceptable standard Time: specify min/max time required to complete the task Realistic: Time, accuracy and completeness criteria allow for successful completion 11
https://www.aretezoe.com/
Develop test items Analyze Learning Objective
•Did the student acquire the required KSAs? •Criterion-based test items •Knowledge-based or performance-based test •Conditions & Scenarios •How well is the test performed
07/FEB/2018
Determine type of test item
•Performance-based: •Subject knowledge & the ability to perform • Assess the process • Assess the product •Knowledge-based tests • True/false • Multiple choice • Matching • Listing • Fill-in-the-blank • Short answer • Labeling • Essay
Write test items
Record test items
•Test construction occurs in the DEVELOP phase
•Learning Objective Worksheet (LOW)
•Performance objective •List steps, activities, behaviors (process) & parameters (product) •Arrange & reorder steps/activities •Review the checklist •Instructions for the evaluator •Instructions for the student •Performance checklist
•Master Lesson File (MLF) •Enter test items in training management system for automated grading
12
https://www.aretezoe.com/
Demonstration
Self-study (reading)
Self-paced (computer-based)
Questioning (Socratic method)
Non-directed discussion
Guided discussion
Practical application (individual or group)
Field trips
Simulations (roleplaying, games)
Case study
Coaching
Application
Indirect discourse (Panel discussion)
Interaction
Lecture
Presentation
Higher levels
X
X
X
X
OK
OK
X
X
HR
X
HR
HR
HR
Comprehension
HR
HR
X
OK
HR
HR
X
HR
OK
X
OK
OK
X
Knowledge
HR
OK
X
HR
HR
X
X
X
OK
OK
X
X
X
Higher level
X
X
HR
X
OK
X
X
X
HR
X
OK
X
HR
Lower level
X
X
HR
X
OK
X
X
X
HR
X
OK
X
HR
Higher level
X
X
X
X
X
X
X
X
X
OK
HR
HR
X
Lower level
HR
HR
X
OK
OK
OK
OK
HR
OK
HR
HR
OK
HR
X – Not Recommended OK – Recommended HR – Highly Recommended
Cognitive domain
Psychomotor
Affective
07/FEB/2018
13
https://www.aretezoe.com/
Sequence TLO Ensures efficiency, proper transition and structured learning RELATIONSHIPS Grouping TLOs •
Same object
•
Same condition
Relationships in sequencing •
Dependent
•
Supportive
•
Independent
•
Conflicting
07/FEB/2018
SEQUENCING STEPS
Group TLOs based on shared elements
Determine relationships between TLOs
Arrange TLOs based on their relationships
DESIGN PHASE OUTPUTS Target Population Description Learning Objectives (ELO, TLO)
Test items Methods and Media Sequenced TLOs
14
https://www.aretezoe.com/
Do the knowledge & skills accurately reflect job needs?
Do the LOs support individual training standard?
Is the LO accurately defined?
Does the test accurately measure the knowledge, skill or task?
Are the testing methods appropriate?
Do the assessment instruments accurately measure competence?
Is the material organized logically? 07/FEB/2018
Do the test items consistently measure knowledge or performance?
Is the delivery system appropriate?
Use checklists to record evaluation
EVALUATE Document Review
Analyze Phase Evaluation
Design Phase Evaluation
Develop Phase Evaluation
Implement Phase Evaluation
Analyze & Interpret Data Manage Evaluation Data Course Content Review
15
https://www.aretezoe.com/
Target Population Description (TPD), Learning Analysis (LA) and sequenced Learning Objectives (ELO, TLO) become the source documents for the DEVELOP phase
DEVELOP Course structure
Lesson Materials
Course Descriptive Data (CDD)
•
Purpose
•
Secure resources
•
Steps
•
Write lesson plan
•
Materials & media
Program of Instruction (POI)
Instructor guide
•
Detailed description of the course
•
Plan for satisfying requirements outlined in Individual Training Standard or Manual
Concept cards
•
Purpose
•
•
Categories
Construct tests
•
Elements
•
•
Methods, categories
Operational Risk Assessment
Validating instruction
Structure
Materials
Summary of resources required to administer a course
Assemble Master Lesson File (MLF) •
1 MLF for every class
Outputs
Course Structure
Lesson materials
Course Descriptive Data (CDD)
Concept Cards
Tests
Program of Instruction (POI)
Operational Risk Assessment
Validation
Master Lesson File (MLF)
07/FEB/2018
16
https://www.aretezoe.com/
Concept Cards Course
title: First aid
Lesson/exam Hours
title and ID: CPR / ID T01-007
per method: 1 hr lecture / 2 hrs practice
Student
: instructor ratio: 1 instructor, 1 assistant, 4-6 students
Media: Lecture Learning
and practical demonstration
objectives: Perform CPR until ambulance arrival
Lesson
purpose: Learn to provide competent first aid in workplace prior to ambulance arrival Material requirements:
Defibrillator, dummy, projector
Exam:
Practical demonstration, a case study
Notes:
external instructor
References:
CPR manual, defibrillator instructions for use
Signatures:
Authorized person
07/FEB/2018
17
https://www.aretezoe.com/
Operational Risk Assessment Hazard identification
• Gather resources: MLF, manuals, SOPs mishap reports Subject matter expert’s input • Identify steps • Training vs. job
07/FEB/2018
Assess hazards
Make risk decisions
• Assess severity
• Assess probability • Assign RAC - Risk Assessment Code
• Prioritize high-risk, high-probability events • Control measures • Benefit:risk, cost
Implement controls Supervise • Record instructions • Implementation plan • Accountability & support • Cost:benefit • Document changes
18
https://www.aretezoe.com/
Operational Risk Assessment Hazard assessment
Probability of occurrence
Probability of hazard occurrence A – likely, frequently (individual)/ continuously (group) B – probably, several times / frequently C – can reasonably be expected to occur D – unlikely to occur
07/FEB/2018
Severity
Severity of hazard I – death, loss of facility/asset, grave damage to business interest II – severe injury/illness, property damage, damage to business interest, degradation of efficient use of assets III – minor injury/illness, property damage, damage to business interest, degradation of efficient use of assets IV – minimal threat
Risk Matrix
Asses severity and probability of hazard and assign Risk Assessment Code
A
B
C
D
I
1
1
2
3
II
1
2
3
4
III
2
3
4
5
IV
3
4
5
5
Risk Assessment Code
1 – Critical 2 – Serious 3 – Moderate 4 – Minor 5 – Negligible 19
https://www.aretezoe.com/
Lesson materials MLF binder
Secure resources
Instructor preparation guide
07/FEB/2018
Student outlines
Lesson plans
Learning Objective Worksheet (LOW)
Lesson Plan: • Provides a detailed script • Establishes continuity • Ensures standardization • Record keeping
Student material
Media
Learning Analysis Worksheet (LAW)
20
https://www.aretezoe.com/
Validating instruction A process of trying the material prior to implementation and examine its effectiveness.
Methods
Collected Data
Steps Check SOPs
Review curriculum for objectivity Review technical data for correctness Review for appropriateness for target audience Pilot course
07/FEB/2018
Student feedback
Schedule revisions
Effectiveness of materials
Determine data collection procedures and techniques
Effectiveness of delivery systems Reliability of tests
Conduct validation Validation report Make revisions
21
https://www.aretezoe.com/
Outputs of DEVELOP Phase Course Descriptive Data
Program of Instruction
Master Lesson File
How does the concept meet organization’s training needs
Management tool for conducting a course
File that contains everything needed to conduct a lesson
Resource requirements
Consolidation of materials from previous phases
Responsibilities Purpose & Scope Length & Breakdown Capacity Student prerequisites Funding
Certification
Record of changes Course Descriptive Data
•
Learning analysis worksheet
•
Learning objectives worksheet
•
Concept card
•
Operational risk assessment worksheet
•
Instructor preparation guide
•
Lesson plan
•
Student outline
•
Supplemental student materials
•
Media
Summary of hours Concept cards Student evaluation Distribution list
07/FEB/2018
Living document, continuously updated
22
https://www.aretezoe.com/
Is the content consistent?
Do the materials support the LOs?
Does the method facilitate maximum learning?
Is the method appropriate to the subject matter?
Are training aids suitable?
Is the instructional approach current?
Validate checklists 07/FEB/2018
Are examples, exercises and feedback realistic and accurate?
EVALUATE Document Review
Analyze Phase Evaluation
Design Phase Evaluation
Develop Phase Evaluation
Implement Phase Evaluation
Analyze & Interpret Data
Is there enough time for instruction and practice?
Manage Evaluation Data
Expert review
Course Content Review
23
https://www.aretezoe.com/
IMPLEMENT Delivery of instruction takes place in the implementation phase. Conduct instruction - Communication
Prepare & rehearse
Review lesson material - Personalize lessons - Schedule
- Student materials - Tests
07/FEB/2018
Time-critical Operational Risk Assessment - Identify changes - Change analysis
- Prepare setting, media and student materials - Coordinate with staff
- Administration
- Questions - Relevance, focus & motivation - Multiple techniques
- Rehearsals - Instructor evaluation
- What if Tool - 5-step ORM process
- Verbal & non-verbal
Administer student evaluations
After lesson management - Assess the effectiveness of instruction
- Recommendations for changes
24
https://www.aretezoe.com/
EVALUATE
Course materials
Do materials support LOs? Are they suitable & easy to follow? Are the test instructions clear?
Document Review
Evaluating Instructor Is the instruction effective?
Positive feedback to students? Is test administration effective?
Analyze Phase Evaluation
Design Phase Evaluation
Develop Phase Evaluation
Implement Phase Evaluation
Evaluating environment
Does the setting facilitate maximum learning? Is the instructor : student ratio adequate?
Analyze & Interpret Data Manage Evaluation Data
Evaluating student performance Are students mastering the LOs? Are students able to perform tasks?
Course Content Review
Any problem areas? 07/FEB/2018
25
https://www.aretezoe.com/
OVERALL COURSE EVALUATION Are graduates able to perform the job successfully? •
Multiple tools available to evaluate instructors, students, and the course
•
Forms, checklists and interviews
•
Descriptive statistics
•
Graduate performance
Document Review
Analyze Phase Evaluation
Design Phase Evaluation
Develop Phase Evaluation
Implement Phase Evaluation
Analyze & Interpret Data Manage Evaluation Data Course Content Review
07/FEB/2018
26
https://www.aretezoe.com/
ARETE-ZOE, as a consultancy, provides solutions to complex problems in the high stakes and high consequence environment of Global Pharmaceuticals, including clinical research, healthcare informatics, and public health. We blend established, Pharma sector methodologies, innovation, and adaptations/transfers from other sectors to identify and resolve consequential practices that pose risk and often result in avoidable patient casualty. However, we are specifically, not a patient advocacy group but believe in optimizing organizational effectiveness and that smart business is agile, competitive and profitable, while intrinsically safe, secure, and resilient. We work within a global context because transnational interests influence national circumstances and choices at point of prescription.
ARETE-ZOE, provides full spectrum organizational and operational risk management consultancy. Our published materials provide a glimpse of some aspects of our services to demonstrate both knowledge and ongoing participation within the Pharmaceutical Industry. Our analysis and consultancy includes all channels of misuse, diversion, counterfeiting and illicit exploitation of pharmaceuticals, medical devices, and precursor chemicals. Our advisement is to manufactures, jurisdictional entities, insurers, legislators, litigators, patients, and health care providers. This scope also frequently segues into the nexus of crime and terrorism as significant influencers that undermine sector integrity differentiated from other criminal activity. Obviously, vulnerability assessment, information collection management and intelligence production supporting decision-making for risk reduction and interventions are routinely within the scope of our services as well as design and implementation of operational control measures.
07/FEB/2018
27