ChatGPT may help residents improve their understanding of orthopaedic surgery and general orthopaedic principles

Written By : Dr Supreeth D R |Medically Reviewed By : Dr. Kamal Kant Kohli Published On 2024-02-11T22:00:06+05:30 | Updated On 11 Feb 2024 10:00 PM IST

Publicly available AI language models such as ChatGPT have demonstrated utility in text generation and even problem-solving when provided with clear instructions. Amidst this transformative shift Diane Ghanem et al conducted a study to assess ChatGPT's performance on the orthopaedic surgery in-training examination (OITE). Investigation performed at The Johns Hopkins Hospital, Baltimore, Maryland. It has been published in “JBJS Open Access.”

All 213 OITE 2021 web-based questions were retrieved from the AAOS-ResStudy website (https://www.aaos. org/education/examinations/ResStudy). Two independent reviewers copied and pasted the questions and response options into ChatGPT Plus (version 4.0) and recorded the generated answers. All media-containing questions were flagged and carefully examined. Twelve OITE media-containing questions that relied purely on images (clinical pictures, radio graphs, MRIs, CT scans) and could not be rationalized from the clinical presentation were excluded. Cohen's Kappa coefficient was used to examine the agreement of ChatGPT-generated responses between reviewers. Descriptive statistics were used to summarize the performance (% correct) of ChatGPT Plus. The 2021 norm table was used to compare ChatGPT Plus' performance on the OITE to national orthopaedic surgery residents in that same year.

Key findings of the study:

• A total of 201 questions were evaluated by ChatGPT Plus.

• Excellent agreement was observed between raters for the 201 ChatGPT-generated responses, with a Cohen's Kappa coefficient of 0.947. 45.8% (92/201) were media containing questions.

• ChatGPT had an average overall score of 61.2% (123/201). Its score was 64.2% (70/109) on non media questions.

• When compared to the performance of all national orthopaedic surgery residents in 2021, ChatGPT Plus performed at the level of an average PGY3.

ChatGPT Plus is able to pass the OITE with an overall score of 61.2%, ranking at the level of a third-year orthopaedic surgery resident. It provided logical reasoning and justifications that may help residents improve their understanding of OITE cases and general orthopaedic principles. Further studies are still needed to examine their efficacy and impact on long-term learning and OITE/ABOS performance.

MBBS, Dip. Ortho, DNB ortho, MNAMS

Dr Supreeth D R (MBBS, Dip. Ortho, DNB ortho, MNAMS) is a practicing orthopedician with interest in medical research and publishing articles. He completed MBBS from mysore medical college, dip ortho from Trivandrum medical college and sec. DNB from Manipal Hospital, Bengaluru. He has expirence of 7years in the field of orthopedics. He has presented scientific papers & posters in various state, national and international conferences. His interest in writing articles lead the way to join medical dialogues. He can be contacted at editorial@medicaldialogues.in.

Dr. Kamal Kant Kohli

Dr Kamal Kant Kohli-MBBS, DTCD- a chest specialist with more than 30 years of practice and a flair for writing clinical articles, Dr Kamal Kant Kohli joined Medical Dialogues as a Chief Editor of Medical News. Besides writing articles, as an editor, he proofreads and verifies all the medical content published on Medical Dialogues including those coming from journals, studies,medical conferences,guidelines etc. Email: drkohli@medicaldialogues.in. Contact no. 011-43720751