Patients to be guarded about ChatGPT as potential resource until its credibility to provide information is verified

Rapid technological advancements have laid the foundations for the use of artificial intelligence in medicine. The promise of machine learning (ML) lies in its potential ability to improve treatment decision making, predict adverse outcomes, and streamline the management of perioperative healthcare. In an increasing consumer-focused health care model, unprecedented access to information may extend to patients using ChatGPT to gain insight into medical questions.

J.A. Dubin et al conducted a study to replicate a patient’s internet search in order to assess the appropriateness of ChatGPT, a novel machine learning tool released in 2022 that provides dialogue responses to queries, in comparison to Google Web Search, the most widely used search engine as a resource for patients for online health information.

For the 2 different search engines, the authors compared

i) the most frequently asked questions (FAQs) associated with total knee arthroplasty (TKA) and total hip arthroplasty (THA) by question type and topic; ii) the answers to the most frequently asked questions; as well as

iii) the FAQs yielding a numerical response.

A Google web search was performed with the following search terms:

“total knee replacement” and “total hip replacement.” These terms were individually entered and the first 10 FAQs were extracted along with the source of the associated website for each question.

The following statements were inputted into ChatGPT:

1) “Perform a google search with the search term ‘total knee replacement’ and record the 10 most FAQs related to the search term” as well as

2) “Perform a google search with the search term ‘total hip replacement’ and record the 10 most FAQs related to the search term.”

A Google web search was repeated with the same search terms to identify the first 10 FAQs that included a numerical response for both “total knee replacement” and “total hip replacement.” These questions were then inputted into ChatGPT and the questions and answers were recorded.

Key findings of the study were:

• There were 5 of 20 (25%) questions that were similar when performing a Google web search and a search of ChatGPT for all search terms.

• Of the 20 questions asked for the Google Web Search, 13 of 20 were provided by commercial websites.

• For ChatGPT, 15 of 20 (75%) questions were answered by government websites, with the most frequent one being PubMed.

• In terms of numerical questions, 11 of 20 (55%) of the most FAQs provided different responses between a Google web search and ChatGPT.

The authors concluded that – “A comparison of the FAQs by a Google web search with attempted replication by ChatGPT revealed heterogenous questions and responses for open and discrete questions. ChatGPT should remain a trending use as a potential resource to patients that needs further corroboration until its ability to provide credible information is verified and concordant with the goals of the physician and the patient alike.”

Further reading:

Using a Google Web Search Analysis to Assess the Utility of ChatGPT in Total Joint Arthroplasty

J.A. Dubin et al.

The Journal of Arthroplasty 2023

https://doi.org/10.1016/j.arth.2023.04.007