Gordon 2006 simulationbasedteachingvstraditional

Advances in Health Sciences Education (2006) 11: 33–39 DOI: 10.1007/s10459-004-7346-7

! Springer 2006

A Randomized Controlled Trial of SimulationBased Teaching versus Traditional Instruction in Medicine: A Pilot Study among Clinical Medical Students JAMES A. GORDON1,2,3,*, DAVID W. SHAFFER4, DANIEL B. RAEMER1,5, JOHN PAWLOWSKI1,2,6, WILLIAM E. HURFORD1,5,7, JEFFREY B. COOPER1,5 1

Center for Medical Simulation, Boston; 2G.S. Beckwith Gilbert and Katharine S. Gilbert Medical Education Program in Medical Simulation, Harvard Medical School; 3Department of Emergency Medicine, Massachusetts General Hospital; 4Department of Educational Psychology, University of Wisconsin-Madison; 5Department of Anaesthesia and Critical Care, Massachusetts General Hospital; 6Department of Anesthesia and Critical Care, Beth Israel Deaconess Medical Center; 7 Department of Anesthesia, University of Cincinnati (*Corresponding author: Department of Emergency Medicine, Massachusetts General Hospital, 55 Fruit Street, Clinics 115, Boston, MA 02114; Phone: 617-726-7622; Fax: 617-724-0917; e-mail: jgordon3@partners.org) (Received 25 March 2004; accepted 7 December 2004)

Abstract. Objective: To compare simulator-based teaching with traditional instruction among clinical medical students. Methods: Randomized controlled trial with written pre-post testing.

Third-year medical students (n ¼ 38) received either a myocardial infarction (MI) simulation followed by a reactive airways disease (RAD) lecture, or a RAD simulation followed by an MI lecture. Results: Mean pre-post test score improvement was seen across teaching modalities (overall change score [simulation]=8.8 [95% CI ¼ 2.3–15.3], pretest [62.7]; change score [lecture] ¼ 11.3 [95% CI ¼ 5.7–16.9], pretest [59.7]). However, no significant differences were observed between simulator-based teaching and lecture, in either subject domain. Conclusions: After a single instructional session for clinical medical students, differences between simulatorbased teaching and lecture could not be established by the written test protocols used in this pilot.

Future studies should consider the effects of iterative exposure assessed by clinical performance measures across multiple centers.

Introduction Experiential learning, or ‘‘learning by doing,’’ (Dewey, 1916) has long dominated the culture of professional apprenticeships. In medical training, however, this powerful pedagogical technique carries an inherent risk to patients. ‘‘See one, do one, teach one’’ has long been a guiding principle of medical trainees, yet leaves precious little room for error (Freidrich, 2002).

While ‘‘safe’’ modalities of lecture and small-group teaching are essential to

34

JAMES A. GORDON ET AL.

the educational process, they cannot replace the value of clinical experience.

The advent of realistic medical simulation, however, promises to enhance the apprenticeship of young physicians without any risk to patients. These sophisticated robot-mannequins can replicate the clinical encounter with such a degree of realism that trainees can intensively ‘‘practice’’ medicine in a safe environment in preparation for actual patient care (Gaba et al., 2001; Gordon et al., 2004).

Despite its inherent appeal, however, objective evidence for the efficacy of simulator-based medical education remains limited (Issenberg et al., 1999; Morgan et al., 2002). We conducted this study to compare simulator-based teaching with traditional instruction in critical care and emergency medicine among clinical medical students. Our hypothesis was that a single exposure to realistic patient simulation would help students to integrate basic and clinical science in a way that enhances trainee understanding compared with traditional teaching.

Methods STUDY

DESIGN

We conducted a randomized controlled trial of simulation based-teaching versus traditional instruction among clinical medical students, using pre-post written evaluations (n ¼ 38). Two subject domains (reactive airways disease [RAD] and myocardial infarction [MI]) were ‘‘crossed over’’ in the intervention phase, producing two experimental cohorts: (1) students receiving a respiratory simulation followed by a cardiac lecture [n ¼ 17], and (2) students receiving a cardiac simulation followed by a respiratory lecture [n ¼ 21]. We hypothesized that for each subject domain, students who received simulation training would outperform students who received lecture training, across cohorts.

SETTING

AND POPULATION

A convenience sample of all 3rd and 4th year Harvard medical students rotating through a surgery clerkship at Beth Israel Deaconess Medical Center were enrolled in the study during the winter–spring of 2000 (n ¼ 38). Each study day accommodated a single group of 3 students, who were randomized as a team to a particular simulation/lecture cohort based on their date of participation.

A single instructor (emergency medicine physician) conducted all simulation exercises (both subject domains) while two different faculty gave the lectures (anesthesiologist gave all respiratory lectures; emergency physician

A RANDOMIZED CONTROLLED TRIAL OF SIMULATION-BASED TEACHING

35

gave all cardiac lectures). While both lectures were considered ‘‘traditional teaching’’ for study purposes, the cardiac lecture was a more of a traditional didactic (as if given in an auditorium full of students), while the respiratory lecture became more of a tutorial discussion.

All study sessions were conducted at the Center for Medical Simulation in Boston, and used the Eagle high-fidelity patient simulator (MedSim-Eagle Simulation, Inc.).

PROTOCOL/INTERVENTION On the appointed day, each group of 3 subjects presented to the simulation center for, in order: (1) a 30-minute written pretest [both subject domains]; (2) a 90-minute simulation exercise [single subject domain]; (3) a 90-minute lecture [complementary subject domain]; and (4) 30-minute post-test [both subject domains].

An instructional ‘‘challenge model’’ was designed to compare the two teaching modalities. All instructors were given only the following instructions to guide their instructional plan: ‘‘You will have 90 minutes to teach a group of 3 students about either reactive airways disease or myocardial infarction.

After each session the student should: (1) understand the clinical care of the relevant disease process, (2) understand the basic physiology and pharmacology of the care process, (3) appreciate the complications of the disease and care process, and (4) communicate effectively about the encounter.’’ No further guidance was given. Instructors were told to use their respective modalities (simulation vs. lecture) as they saw fit, to achieve the outlined objectives for the assigned subject domain. All instructors were blinded to the other’s specific teaching materials and lesson plan; they were also blinded to the content of the written tests (see below).

MEASUREMENTS Based only on the global learning objectives, two equivalent written tests in both subjects (cardiac version A or B; respiratory version A or B) were written and graded by an expert blinded to the specific instructional plans (intensivist-anesthesiologist). For each study day, the versions A and B (for both subject domains) were randomized as pre-test or post-test. Each test version comprised 11 or 12 short-answer/multi-part questions, and was worth 100 points. The tests were piloted among students and refined in a pre-study phase based on the same protocol.

A typical exam might begin with a physical diagnosis question: ‘‘Describe the physical signs and symptoms that may be suggestive of an acute myocardial infarction (or asthma attack)’’ [10 points]. It might then probe

36

JAMES A. GORDON ET AL.

Table I. Changes in pre-post written test performance (mean score) for simulation-based teaching versus traditional instruction among clinical medical students, stratified by subject domain (reactive airways disease [RAD], myocardial infarction [MI])

Subject Simulation domain N Pre (SD)

Lecture1 Post (SD) Change (SD) N Pre (SD)

RAD 17 63.4 (14.7) 69.8 (16.4) 6.5 (15.9) MI 21 62.2 (17.9) 73.0 (10.8) 10.8 (22.6) Overall 38 62.7 (16.3) 71.6 (13.5) 8.8 (19.8) 1

Post (SD) Change (SD)

21 61.0 (17.8) 71.5 (14.4) 10.5 (15.9) 17 58.2 (15.7) 70.5 (16.3) 12.2 (18.6) 38 59.7 (16.7) 71.0 (15.1) 11.3 (16.9)

The RAD ‘‘lecture’’ evolved into a small group tutorial session during the study.

understanding of management and complications: ‘‘When would you consider endotracheal intubation and mechanical ventilation (in a refractory asthmatic)?’’ [5 points]; or, ‘‘When would you not want to use thrombolytic therapy in a patient with an acute MI?’’[5 points]. The test might conclude: ‘‘After successful treatment, the patient and family ask how they might reduce risk for a subsequent event – what would you tell them?’’ [10 points].

DATA

ANALYSIS

For each subject domain, data were analyzed by analysis of covariance (ANCOVA). Post-test score was designated as the dependent outcome variable, controlled by pre-test score, intervention format (lecture vs. simulation), pre-post test sequence (version A then B vs. version B then A), and gender. Significance was set at <0.05 for the F statistic in the ANCOVA model.

Results Mean pre-post test score improvement (Table I) was seen across teaching modalities (overall change score [simulation] ¼ 8.8 [95% CI ¼ 2.3–15.3], pretest [62.7]; change score [lecture] ¼ 11.3 [95% CI ¼ 5.7–16.9], pretest [59.7]). For cohort 1, average improvement ranged from 6.5 test points for the respiratory simulation (95% CI [)1.7–14.6], pretest [63.4]) to 12.2 test points for the myocardial infarction lecture (95% CI [2.6–21.8], pretest [58.2]). For Cohort 2, average improvement ranged from 10.5 for the respiratory lecture (95% CI [3.3–17.7], pretest [61.0]) to 10.8 for the cardiac simulation (95% CI [0.50–21.1], pretest [62.2]). While both cohort groups showed mean score improvement across both subject domains, no significant differences were observed in comparative magnitude of pre-post score

A RANDOMIZED CONTROLLED TRIAL OF SIMULATION-BASED TEACHING

37

changes. Neither intervention format (lecture vs. simulation) nor gender were significant predictors of improvement in the multivariate model.

Discussion In this population of clinical medical students, we did not detect any significant differences in subject performance based on teaching modality.

Although we hypothesized a differential effect, our results are consistent with decades of research on the comparative effects of technology-enhanced learning (i.e., distance education) (Russel, 1999). In demonstrating ‘‘no significant difference’’ between traditional and technology-enhanced teaching, novel approaches are often labeled ‘‘equivalent’’ to the established paradigm (Twigg, 2001). However, caution must be used in interpreting the results of such studies. In our study, each group experienced only one intervention for each subject domain, a brief exposure that may be inadequate to detect differences in teaching effect. Given the large variance we observed among some scores, even a doubling of our sample size would yield just 80% power to detect a 10-point differential between intervention arms [SD ¼ 15]). In addition, each of our study days only comprised 3 students, creating a ‘‘small group’’ dynamic regardless of the intervention format (the respiratory ‘‘lecture’’ actually became more of a tutorial session). Comparative differences could thus be obscured – or exaggerated – by random differences in small group interactions.

Even if ‘‘no significant difference’’ translated into true equivalence among educational approaches, adoption of new technology is often limited by cost (original simulators cost nearly $200,000; newer models now sell for under $50,000). Novel approaches must offer compelling advantages. While simulators may in fact promote safe, standardized, and efficient learning, a randomized controlled trial may not be the best approach to studying educational outcomes (Prideaux, 2002). In our own attempt at an educational RCT, we were clearly limited by strict adherence to a blinded protocol, which proved very challenging logistically. Despite the circulation of broad learning and teaching objectives, study personnel were so intensively blinded to each others’ activities that it became difficult to ensure a reliable correlation between instructional material and testing instruments. Moreover, scheduling constraints precluded us from randomizing the daily order of events, so that each simulator session was followed by a lecture given right before the posttest. This may have fostered enhanced retention and clarity of lecture material simply because it was ‘‘fresh’’ right before the test.

Although we did explore qualitative interviews and a simulator-based evaluation exercise, we chose to use a written assessment as the primary outcome measure because we felt the quantitative metrics could be more

38

JAMES A. GORDON ET AL.

reliably assessed. In retrospect, we suspect that simulation may actually promote the rapid translation of knowledge (‘‘knows’’) into reasoned action (‘‘does’’), a leap of cognition that would be poorly assessed by written testing alone (Maatsch, 1981; Schuwirth and van der Vleuten, 2003; Miller, 1990).

Future use of clinical performance assessment tools (Gordon et al., 2003; Tekian et al., 1999) may help better define the relationship between learner enthusiasm for this technology and efficacy of its use.

Conclusion After a single instructional session for clinical medical students, differences between simulator-based teaching and lecture could not be established by the written test protocols used in this pilot. Future studies should consider the effects of iterative exposure assessed by clinical performance measures across multiple centers.

Acknowledgements This study was funded by a grant from the Josiah Macy, Jr. Foundation to the Harvard-MIT Division of Health Sciences and Technology and the Center for Medical Simulation. Dr. Gordon is a member of the Clinical Education Advisory Board for Medical Education Technologies, Inc., without personal compensation; all proceeds are donated to Harvardaffiliated teaching programs. The authors would like to thank Erin GraydonBaker, George Topulos, David Tancredi, Martha Gray, Mark D’Avila, David Feinstein, and the staff of the Center for Medical Simulation in Boston for their contributions to this work.

References Dewey, J. [1916](1994). Thinking in education. Reprinted in: L.B. Barnes, C.R. Christensen & A.J. Hansen (eds), Teaching and the Case Method, 3rd ed. (pp. 9–14). Boston: Harvard Business School Press.

Freidrich, M.J. (2002). Practice makes perfect: Risk-free medical training with patient simulators JAMA 288: 2808–2812.

Gaba, D., Howard, S., Fish, K., Smith, B. & Soub, Y. (2001). Simulation in anesthesia crisis management: A decade of experience. Simulation and Gaming. 32(2): 175–193.

Gordon, J.A., Oriol, N.E. & Cooper, J.B. (2004). Bringing good teaching cases ‘‘to life’’: A simulator-based medical education service. Academic Medicine 79(1): 23–27.

Gordon, J.A., Tancredi, D.N., Binder, W.D., Wilkerson, W.M. & Shaffer, D.W. (2003).

Assessment of a clinical performance evaluation tool for use in a simulator-based testing environment: A pilot study. Academic Medicine (October supplement) 78(10): S45–S47.

A RANDOMIZED CONTROLLED TRIAL OF SIMULATION-BASED TEACHING

39

Issenberg, S.B., McGaghie, W.C., Hart, I.R., et al. (1999). Simulation technology for health care professional skills training and assessment. JAMA 282(9): 861–866.

Maatsch, J.L. (1981). Assessment of clinical competence on the emergency medicine specialty certification examination: The validity of examiner ratings of simulated clinical encounters.

Annals of Emergency Medicine 10: 504–507.

Miller, G.E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine 65(Suppl. 9): S63–S67 Morgan, P.J., Cleave-Hogg, D., McIlroy, J. & Devitt, J.H. (2002). A comparison of experiential and visual learning for undergraduate medical students. Anesthesiology 96: 10–16.

Prideaux, D. (2002). Researching the outcomes of educational interventions: A matter of design [editorial]. British Medical Journal 324: 126–127.

Russel, T.L. (1999). The No Significant Difference Phenomenon. Montgomery, Alabama: International Distance Education Certification Center (IDEC).

Schuwirth, L.W. & van der Vleuten, C.P. (2003). The use of clinical simulations in assessment.

Medical Education 37(Suppl. 1): 65–71.

Tekian, A., McGuire, C.H., McGaghie, W.C., et al. (1999). Innovative Simulations for Assessing Professional Competence: From Paper and Pencil to Virtual Reality. Chicago: Department of Medical Education, University of Illinois at Chicago, 1999.

Twigg, C.A. (2001). Innovations in Online Learning: Moving Beyond No Significant Difference.

Troy, New York: The Pew Technolgy and Learning Program at the Center for Academic Transformation, Rensselaer Polytechnic Institute. http:// www.center.rpi.edu/PewSym/ Mono4.pdf. Accessed November 23, 2004.