IN FECTIO N CO NTRO L
Using Machine Learning to Predict Surgical Site Infections Results of Two Studies Conflict on Generalizability of Algorithms
By MONICA J. SMITH
(GLM) in predicting colorectal deep organ-space SSIs (C-OSIs, hile most health systems are at the beginning of the jour- e.g., postoperative intraabdominal abscess). ney in using artificial intelligence and machine learning Among the 2,376 elective colorectal resections performed at to predict surgical complications, surgeons at the forefront of this Mayo Clinic between 2006 and 2014, the C-OSI rate was 4.6% science are expanding our knowledge by investigating ways to (108). The BPMI model identified 57 of these patients: a senovercome the obstacles to AI, as described in two recently pub- sitivity of 56%, compared with the GLM’s sensitivity of 47%. lished studies. The BPMI model lost its advantage when the model was built to “These articles pose different questions, but what they have use extra-institutional data (i.e., based on the American College in common is that they’re both examining the challenges of of Surgeons National Surgical Quality Improvement Program), bringing AI to bear on the problem of surgical site infection,” which reduced its sensitivity to 47%. commented Philip S. Barie, MD, MBA, a professor emeritus They concluded that for optimal performance, the BPMI model of surgery at Weill Cornell Medicine, in New York City, and should be built using “data specific to the individual institution” the executive director of the Surgi(Surg Infect 2021;22[5]:523-541). cal Infection Society Foundation for ‘My concern is that somebody will “We’re going to be seeing more Education and Research. and more of these models, and peodevelop a big model based on a very “Studies of SSI prevalence are ple need to understand the limitachallenging to perform and inter- heterogeneous data set that may not tions of them, and how to use them pret if not done prospectively, using in their institution,” Dr. Cima said. reflect the risk profile or the patient trained observers inspecting each “My concern is that somebody incision,” Dr. Barie told OR Man- profile of an individual hospital. I’d hate will develop a big model based on agement News. “Retrospective studies a very heterogeneous data set that to see them penalized or made to look always leave doubt as to what exactmay not reflect the risk profile or the ly was observed, whether patients like they’re not performing well when patient profile of an individual hoswere omitted inadvertently because pital. I’d hate to see them penalized the model was never designed to be of sporadic reporting from the outor made to look like they’re not perpatient setting, or if data reporting is used in their environment.’ forming well when the model was incomplete. Moreover, thousands of never designed to be used in their —Robert Cima, MD patients are required to achieve adeenvironment,” Dr. Cima said. quate statistical power to study clean Because retrospective chart review operations owing to the low prevalence of infection.” is cumbersome, other investigators have sought to automate the One study, conducted by researchers at Mayo Clinic in Roch- process using machine learning and natural-language processing. ester, Minn., addressed the problem of missing data, which can The other study, which specifically investigated the generalizskew retrospective analyses and subsequent prospective predic- ability of SSI-detection machine learning–generated algorithms, tions of SSIs. found that machine learning models designed at one center “The nice thing about machine learning is that it allows the worked just as well at another. system to refine a model as it evolves, as long as you can get data “We’re at the beginning of an acceleration of having machine for the system to look at; we wanted to know what the impact learning and AI used more widely in health care, but the work of missing data is on the ability to model infections,” said Rob- to validate models isn’t always done optimally. In many instancert Cima, MD, a professor of surgery at Mayo Clinic College of es, we expect it to be like a ‘plug-and-play’ technology, where you Medicine and Science, in Rochester, Minn. install the solution in and it works. But the truth is, in some cases “What we found is that unless you do certain corrections, your there is a degradation in performance or the need for more optimodel is going to suffer from it.” mization,” said Genevieve Melton-Meaux, MD, PhD, a profesTo evaluate a method for handling missing data, Dr. Cima and sor of surgery and Institute for Health Informatics core faculty his colleagues compared a Bayesian-Probit regression model with at the University of Minnesota Medical School, in Minneapolis. continued on page 13 multiple imputation (BPMI) with a generalized linear model
W
4
OR Management News • Volume 16 • December 2021
