Study finds ChatGPT to be pretty accurate for common allergy myths

USA: In a new study, ChatGPT was asked to rate 10 common allergy myths, and it judged their accuracy correctly 91% of the time. The study findings were presented at the American College of Allergy, Asthma and Immunology (ACAAI) Annual Scientific Meeting in Anaheim, California. 

Artificial intelligence language models such as ChatGPT are increasingly being used in many different professions, and medicine is not an exception. 

“We searched the PubMed and Google Scholar databases to identify all articles on common allergy myths between 2000-2023,” says Ricardo Estrada-Mendizabal, MD, ACAAI member and lead author of the study. “Our goal was to assess the accuracy of ChatGPT in correctly identifying common allergy myths. We know allergy and asthma patients are constantly exposed to misinformation on the internet. We thought that the surge of public language models could represent a bigger threat to misinformation spread, especially because ChatGPT is unable to provide references.”

Ten myths were selected and entered in ChatGPT 4.0 with the prompt “true or false.” ChatGPT answers were graded by 24 allergists on a Likert scale of 1-4 (1=inaccurate [100% not true], 2=somewhat inaccurate [>30% not true], 3= somewhat accurate [<10% not true], 4=accurate [100% true]). Finally, allergists were asked their likelihood of using ChatGPT for patient education with a Likert scale of 1-5 (1=not likely at all–5=very likely). Overall, ChatGPT had an accurate or somewhat accurate response in 91% of cases and an inaccurate or somewhat inaccurate response in 9%.

Said Dr. Estrada-Mendizabal, “The response to whether a positive allergy test indicates a clinical allergy was the most accurate response (96% accurate), whereas the response to if hypoallergenic animals were better for people with asthma and allergies was the most inaccurate (17% inaccurate and 21% somewhat inaccurate response). Eighteen allergists (75%) were open (likely or very likely) to using ChatGPT in the future for patient education.”