Simulation: How do we train in the future? The importance of metrics and standardisation Prof. Dr. Anthony Gallagher ORSI Academy Melle (BE)
ag.gallagher@ ogcmetrics.com “The art of progress is to preserve order amid change and to preserve change amid order.”–Alfred North Whitehead (1861−1947). From the series Great Ideas of Western Man. The acquisition, maintenance and application of skill in surgery and procedure-based medicine are matters of great importance. There is now continued debate and public interest on methods that may be used for quality assurance of surgical performance. Agents of change: Traditionally, procedural skills have been acquired in a formal, structured apprenticeship training system. This was based on a model developed by William Stewart Halsted at Johns Hopkins at the end of the 19th and beginning of the 20th century. The apprenticeship phase in surgery and procedural-based medicine is unlike others because the trainees are being prepared to carry out interventional procedures on sick patients which are frequently life-threatening and almost always pose some risk of morbidity and mortality. There is now an accumulating body of evidence which suggests that the safety of the procedure is directly correlated to the skill of the operator. [1,2] Furthermore, reduced work hours and changing work practices, e.g., image guided interventional procedures (laparoscopic, robotic, endovascular, endoscopic, etc) make the flaws in this approach to training very apparent.
procedure performance. To utilise simulations for training, surgeons had first to develop surgical procedure templates (for a reference approach), including for example; the individual steps of the procedure, and the choice of instruments. They also had to identify optimal and deviations from optimal performance (i.e., errors, critical/sentinel errors) so that engineers and computer scientists could build the simulation and accurately characterise the operation so that performance was quantifiable. The science of simulation training: PBP simulation training is a scientific approach to surgical skills training that is objective, transparent and fair to the trainee as well as the trainer. The performance metrics (i.e., procedure steps, errors and critical/sentinel errors) are the cornerstone of PBP training. These are derived from, validated by, and benchmarked on experienced surgeons who are actually good at performing the procedure in question. The metrics are developed initially during a detailed procedure characterisation with three to five experienced surgeons. [5,6] The metrics explicitly identify observable performance before, during, and after the surgical procedure. They are then validated initially at a Delphi consensus meeting and then construct validity. The latter requires that the metrics can be scored reliably by independent raters (i.e., with an interrater reliability (IRR) >0.8) and the performance assessments reliably discriminate between the objectively assessed performance of experienced and less experienced surgeons. Only when all of these validation criteria are met, a proficiency benchmark can be quantitatively defined, based on the mean performance of the experienced practitioners.
Figure 1a-d: 1a The sculpture at the front of ORSI Academy representing the ambition to scientifically measure performance to augment and enhance robotic surgical skills learning; 1b-d three different ORSI faculty surgeons training robotic surgical skills using the exact same metric-based, deliberate practice curriculum for all trainees.
The leadership of ERUS, the EAU and ORSI Academy are well advanced in this scientific ‘conversation’ and are very aware of the stakes involved. They also know In addition, the performance metrics should be that the scientific method and the data derived from explicit and binary scores, and not Likert scale studies in robotics, endourology, train-the-trainers assessments. Despite the voluminous reports on Likert etc. will guide and underpin their decision-making, scale assessments, they have been demonstrated to thus preserving change amid order. Good quality be unreliable, [7] and thus, by default not valid. PBP scientific data can also mitigate the risk that change Simulation training: validated metrics are then used to give the trainee gets bogged down in endless deliberations. The Halstedian apprenticeship approach to training is explicit formative performance feedback during no longer fit for purpose. It is inefficient, lacks training, thus accelerating their learning using Training must be more than an interesting transparency and assessment is subjective, which can deliberate practice [8] training rather than simply educational experience: be (unfairly) used against the trainee to constrain requiring the trainee to engage in repeated practice. This scientific and evidence-based approach to the training progression or indeed completion. There also acquisition of skills for the operating room relies on seems to be unanimous agreement amongst the Furthermore, PBP training is delivered by faculty who systematic, simulation-based, skills curriculum for different procedural-based disciplines that simulation- know and can score the metrics with an IRR > 0.8 and training and education. [6] It means that surgeons based training is a better way to train. However, there have been taught (in a train-the-trainers course) how (and other health care workers) can be optimally is disagreement on how best to use simulation. There to use the metrics for deliberate rather than repeated prepared for the operating room with their is even more disagreement amongst the different practice. performance benchmarked against other surgeons professional groups as to what constitutes an before performing it in vivo. Research has now shown adequate level of simulation fidelity for it to be useful Deliberate practice and standardisation: that surgeons trained using this approach perform and usable. VR computer and other types of simulation means significantly better and make fewer errors than that surgeons can now learn how to perform specific traditionally trained surgeons. [3,9-12] Effectiveness of simulation training: skills or procedures using the exact same devices, in Quantitative evidence already exists which the same way on simulations. In the past they learned Conclusions: demonstrates that simulation is a better way to train. these skills (and made mistakes) on real patients but Training with metric-based simulation ensures [3] The optimal application of this approach (i.e., on a virtual patient or a simulation they can perform learning to a quantitatively defined performance level proficiency based progression or PBP) has the exact same procedure repeatedly and learn what and greater homogeneity in trainee skill-sets. [6] demonstrated the power of simulation to dramatically not to do, as well as what to do. Evidence from prospective, randomised studies shows improve suturing skills, laparoscopic surgical skills, that a PBP approach to education and training interventional cardiology skills, orthopaedic surgery This type of learning with performance feedback (i.e., produces trainees with skill-sets that are 40-60% skills, and anaesthetist skills for childbirth. deliberate practice) constitutes a very powerful better than trainees using a traditional approach to approach to training that contrasts with the traditional training. These studies also show that trainees who In the next 12-24 months ORSI Academy and ERUS apprenticeship model where performance feedback receive the exact same curriculum but without the will report compelling simulation training data for and learning was much more hit-and-miss. This quantitatively defined performance benchmark robot-assisted procedure skills. A recent systematic scientific and metric-based approach means that perform only marginally better than those receiving review and meta-analysis of published, peer-reviewed, simulation training and proficiency benchmarking traditionally training. [11] prospective, randomised and blinded clinical studies can be standardised and implemented across training showed that a PBP simulation-based approach to centres, the EU and wider afield. Furthermore, the These results clearly demonstrate that simulation training resulted in a 60% reduction in objectively metrics, curriculum and proficiency benchmarks are training is effective for skills acquisition but the assessed performance errors in comparison to a not based on the opinions of a few key opinion simulation training must be more than an interesting quality assured traditional approach. [4] However, leaders but consensus between practicing clinicians educational experience. A PBP approach to training publications on simulation to date have only at formal modified-Delphi meetings. Likewise, may be conceptually and intellectually appealing but demonstrated how superficially simulation is proficiency benchmarks are based on the actual it represents a paradigm shift in how surgeons and understood with scant attention paid to the measured performance levels of practicing clinicians. doctors are educated and trained. [13-17] underlying science of what makes for effective This approach to training necessitates a standardised simulation training. curriculum and systematic and agreed approach to References 1. Curtis NJ, Foster JD, Miskovic D, Brown CS, Hewett PJ, delivering it. Such an approach has the potential to A revolution in computer technology has led to the Abbott S, et al. Association of surgical skill assessment considerably reduce performance heterogeneity by problems faced by surgeons. This same technology with clinical outcomes in cancer surgery. JAMA surgery. ‘trainees’. offers a very powerful training solution. Aviation has 2020;155:590-8. used computer generated virtual reality (VR) 2. Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Order amid change: simulations to train pilots for decades. However, Nunn AR, et al. Surgical skill and complication rates after Agents of change have forced surgery and medicine to unlike aeroplanes and airports with standardised bariatric surgery. N Engl J Med. 2013;369:1434-42. consider how future doctors are optimally prepared features, real patients are all different. Furthermore, 3. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, for safe and effective clinical practice. This will the aviation industry has over decades worked out Bansal VK, Andersen DK, et al. Virtual reality training unavoidably mean a change to the way doctors are precise protocols for dealing with different improves operating room performance: results of a trained. The ‘Scientific Method’ has as stated by aeroplanes, airport terrains and flight scenarios. randomised, double-blinded study. Ann Surg. Whitehead, the capacity to preserve order amid 2002;236:458-63; discussion 63-4. change. Proposals and ideas about training can be Surgery in comparison was very much a craft with 4. Mazzone E, Puliatti S, Amato M, Bunting B, Rocco B, quantitively evaluated in a scientific way with robust individual surgeons applying their own art to Montorsi F, et al. A systematic review and meta-analysis empirical evidence underpinning decisions. 14
EUT Congress News
5.
6.
7.
8.
9.
10.
11.
12.
13.
on the impact of proficiency-based progression simulation training on performance outcomes. Ann Surg. 2021;274:281-9. Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, et al. Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 2005;241:364-72. Gallagher AG, O’Sullivan GC. Fundamentals of surgical simulation; Principles & practices London: Springer Verlag; 2011. Satava RM, Stefanidis D, Levy JS, Smith R, Martin JR, Monfared S, et al. Proving the Effectiveness of the fundamentals of robotic surgery (FRS) skills curriculum: A single-blinded, multispecialty, multi-institutional randomised control trial. Ann Surg. 2019. Ericsson KA, Krampe RT, Tesch-Römer C. The role of deliberate practice in the acquisition of expert performance. Psychol Rev. 1993;100:363-406. Ahlberg G, Enochsson L, Gallagher AG, Hedman L, Hogman C, McClusky DA, 3rd, et al. Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. The American Journal of Surgery. 2007;193:797-804. Van Sickle K, Ritter EM, Baghai M, Goldenberg AE, Huang IP, Gallagher AG, et al. Prospective, randomised, double-blind trial of curriculum-based training for intracorporeal suturing and knot tying. J Am Coll Surg. 2008;207:560-8. Angelo RL, Ryu RK, Pedowitz RA, Beach W, Burns J, Dodds J, et al. A proficiency-based progression training curriculum coupled with a model simulator results in the acquisition of a superior arthroscopic Bankart skill set. Arthroscopy: The Journal of Arthroscopic & Related Surgery. 2015;31:1854-71. Cates CU, Lönn L, Gallagher AG. Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators. BMJ Simulation and Technology Enhanced Learning. 2016;2:1-5. Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, et al. Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 2005;241:364-72.
Due to space constraints, the entire reference list can be made available to interested readers upon request by sending an email to: communications@ uroweb.org.
Saturday, 2 July 14:15 - 18:00 Meeting of the EAU Robotic Urology Section (ERUS) Purple Area, Room Elicium 1
June/July 2022
