National Selection to T&O ST3 posts in England 2020 & 2021

Mark AA Crowther

Mark Crowther is a Consultant T&O Surgeon for North Bristol NHS. Since 2015 he has been a member of and now chairs the National Selection Design Group for recruitment of orthopaedic registrar surgeons. In 2017 he was appointed to the Specialty Advisory Committee and is National Selection Lead for Trauma & Orthopaedic Surgery.

As National Selection Lead on the T&O Specialty Advisory Committee (SAC), I chair the Selection Design Group (SDG) made up of consultants, Training Programme Directors, SAC members and representatives of the British Orthopaedic Trainees Association (BOTA) all experienced in many aspects of training and education with a collective desire to ensure the best potential registrars are chosen for higher surgical training.

We convene several times a year to scrutinise the most recent process using data released by Medical and Dental Recruitment and Selection (MDRS), a body responsible to the Statutory Education Bodies (SEBs), and to improve the following year’s process. We then implement delivery of the process used to select T&O ST3s done on a national basis since 2013. During this time, we have introduced a number of changes and improvements to the interview process undertaken with between 300 and 500 applicants over several days at Elland Road football stadium in Leeds. We have removed the infamous ‘killer’ station, developed the role of simulation with actors and successfully implemented the use of iPads to allow real time scoring which has facilitated early recognition of scoring errors. I took over as Chairman at the end of the 2019 process knowing we had a fair and robust process in which trainees, programme directors and trainers had confidence, in no small part because of the hard work of my predecessors David Large and James Hunter.

When the COVID-19 pandemic struck the MDRS acted swiftly in declaring that there would be no face-to-face interviews for any medical specialty in 2020. We found ourselves in a strong position compared to other specialties given our historical process; our self-assessment form was well established and we knew it correlated well with the overall final score. We made a strong case for the use of the self-assessment score further validated with portfolio review, as a sole selection tool, and also for the appointment to LAT posts given the extraordinary circumstances surrounding the 2020 process. Neither proposal was adopted however due to the desire for uniformity across all medical specialties and the uncertainty of the rapidly changing global pandemic. The decision of MDRS was to appoint to substantive training posts using unvalidated self-assessment scores.

We looked closely at our previous year’s data to carefully decide the ‘appointability’ score, the maximum possible being 32. The decision was a score of 21 which carried a reassuringly low risk of appointing an ‘unappointable’ applicant. With a threshold score of 20, the risk would have increased and at 19 that risk would have increased considerably. We predicted we would have 105 appointable applicants. The 12 questions contained within the selfassessment form were also ranked to be used in a tie-break situation prioritising clinical over academic achievement.

After long-listing there were 444 applications in 2020 and these were ranked according to their unvalidated self-assessment score. After the tie-breakers were applied, adjudication was only required on a single pair of tied applications. On offers-day there was an unintended HEE administrative data processing error leading to a smaller than expected number of offers, but this was recognised quickly, and rectified resulting in 107 offers being made. The Training Programme Directors (TPDs) were rightly conservative in their declaration of ST3 vacancies knowing they may have ST8s whose post-CCT fellowship plans could be disrupted resulting in late requests for training extensions as ‘periods of grace’. Ultimately 96 offers of ST3 posts were accepted.

In autumn 2020 the SDG agreed what should be submitted to MDRS as Plans A & B (ideal and fall-back scenarios) given the decision there would be no in person interviews for 2021. The self-assessment form was adjusted adding 12 months to the banding for maximum scores in question 1 (total time since foundation programme or equivalent) and question 2 (time spent in T&O posts) to ensure no applicant was disadvantaged by the events of the pandemic. We were relieved that our Plan A, validated self-assessment and a 30-minute video interview, was agreed. Having written questions that could be delivered without repetition over five days, and confident we would have assessor panels over such a period using three question per interview station, we wrote to the HEE Selection Lead expressing our desire to interview all applicants in accordance with the normal process in Leeds. For logistical reasons HEE were only able to support 90 interviews over four days hence the maximum of 360 interviews. Following several years of declining numbers, 604 applications were submitted in 2021, the most since the advent of National Selection. After a small number of withdrawals and some long-listing rejections, we had 571 applications.

In February 2021 the SDG met remotely over two days to validate the evidence uploaded by each applicant relating to the 12 questions of the self-assessment form. For quality assurance the first session of the process was performed in pairs before breaking out individually. There was real-time discussion of queries, seeking clarity on inconsistencies and consensus decisions made over the course of the meeting. Evidence relating to 6,852 questions was validated with some scores adjusted up or down and justification notes recorded. An impressive undertaking by dedicated T&O surgeons in their own time. Ten days later, a smaller group reassembled remotely for another day to address the 430 appealed questions, 54% being deemed unsuccessful. This resulted in the final ranked order of the 571 eligible applicants, of which the top 360 were offered video interviews. These were to comprise three ten-minute stations akin to three of the five stations traditionally used in Leeds. The ‘Technical Skills’ and ‘Communication’ (with Actors) stations were felt impractical to deliver virtually and aspects of the latter would naturally be assessed in the remaining ‘Portfolio’, ‘Clinical’ and ‘Prioritisation’ stations.

The constrains of COVID-19 meant this year each applicant could only be seen by three interviewers in the half hour interview with each one asking one question which was independently scored by their two colleagues from their own home or office. We ensured the process was as fair as possible with Quality Assurance involving lay reps and SDG members as ’flies on the virtual wall’ to confirm the delivery of a consistent process. For each of the four days of interviews we ran ten panels, each with three interviewers, each overseen by an HEE administrative staff member and each having intermittent QA presence, with each panel undertaking nine interviews per day. The administrative staff and QA members provided me and other senior SDG members with feedback and we were able to see the scores being submitted in real time. This allowed us to address issues with the interviewers immediately by relaying messages to them via the administrative staff. For example, relating to technical errors in submitting scores or to challenge significant discrepancies in marking when identified. There was never any intention to persuade the assessors to change their scores, merely to ensure that they could justify their scores and that this was recorded accurately. Our interview panels comprised experienced assessors well versed in the selection process and extensively trained, for example relating to equality and diversity, so discrepancies were rare, but occasionally scores were modified in accordance with the marking descriptors.

Over four days we delivered successful remote interviews for 360 applicants and of those only 20 failed to reach the appointability threshold. This is not a concept peculiar to T&O. HEE require every medical specialty to draw a line somewhere and ours is an average score of 3.25 out of 5 for each question plus an average selfassessment score. This left us with 340 appointable applicants to fill 177 posts – a roughly 50% chance of success if interviewed – but successful applicants must rank regions and those with specific geographical preferences may still be disappointed. Regarding geographic preferences, we have made it very clear this year that barring unexpected and compelling reasons, we expect all trainees to receive their training in their accepted Programme.

The SDG is acutely aware of the potential for unconscious bias to influence the outcome of the selection process. Under normal circumstances an applicant can emerge from a station in which they have underperformed, compose themselves and start afresh in the next station with a new pair of interviewers. An applicant could see at least ten assessors in the process and we feel this is preferable to this year’s compromise. In addition, as a craft specialty the public and future employers understandably assume that some assessment of technical skills is undertaken when selecting consultant surgeons of the future and whilst core training does involve such assessment, we strongly believe in the face-to-face validity of a technical skills assessment as part of our ST3 interview process. For these reasons we have appealed to HEE that we should return to face-to-face interviews, assuming government rules allow.