ChatGTP more empathetic toward patient inquiry than physicians: JAMA

The research compared written responses from physicians and those from ChatGPT to real-world health questions. The researchers found that ChatGPT outperformed physicians in providing high-quality, empathetic answers to patient questions. A panel of licensed healthcare professionals preferred the responses of ChatGPT 79% of the time and rated ChatGPT's responses as more empathetic and of higher quality.

The rapid evolution of virtual health care has led to increased patient messages concomitant with more work and burnout in healthcare professionals. AI assistants could aid in creating answers to patient questions by drafting responses that clinicians could review.

John W. Ayers, Qualcomm Institute, University of California San Diego, La Jolla, and colleagues aimed to investigate the ability of an AI chatbot assistant (ChatGPT), released in November 2022, to provide empathetic and quality responses to patient questions. They addressed whether AI chatbot assistants respond to patient questions of comparable empathy and quality to those written by physicians in a cross-sectional study.

The investigators used a nonidentifiable and public database of questions from a public social media forum. They randomly drew 195 exchanges from October 2022, where a verified physician responded to a public inquiry. Chatbot responses were produced by entering the original question into a new session.

A licensed healthcare professional team evaluated the original question along with anonymized and randomly ordered responses from physicians and chatbots. They chose "which response is better". They judged both "the quality of information given" (very good, goof, acceptable, poor, very poor) and "the empathy or bedside manner provided" (very empathetic, empathetic, moderately empathetic, slightly empathetic, and not empathetic). Mean outcomes were ordered on a scale of 1 to 5 and compared between physicians and chatbots.

The study led to the following findings:

• Of the 195 responses and questions, evaluators preferred chatbot responses over physician responses in 78.6% of the 585 evaluations.
• Mean physician responses were remarkably shorter than chatbot responses (52 words vs 211 words).
• Compared to physician responses, Chatbot responses were rated of significantly higher quality.
• The proportion of responses rated as good or very good quality (≥ 4) was higher for chatbots than physicians (chatbots: 78.5%; physicians: 22.1%), resulting in a 3.6 times higher prevalence of good or very good quality responses for the chatbot.
• Compared to physician responses, Chatbot responses were also rated significantly more empathetic.
• Compared to that for physicians, the percentage of responses rated empathetic or very empathetic (≥4) was higher for chatbots (physicians: 4.6%; chatbots: 45.1%), relating to a 9.8 times higher prevalence of empathetic or very empathetic chatbot responses.

"Our findings revealed that a chatbot generated quality and empathetic responses to patient questions in an online forum," the researchers wrote. In clinical settings, further exploration of this technology is warranted, such as using a chatbot to draft responses that physicians could edit.

"Randomized trials could further assess whether using AI assistants might improve responses, decrease clinician burnout, and improve patient outcomes," they conclude.

Reference:

Ayers JW, Poliak A, Dredze M, et al. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. Published online April 28, 2023. doi:10.1001/jamainternmed.2023.1838