ChatGPT AI-Empowered Echo Reports Simplify Findings and Save Time for Patients: Study

Since the implementation of the 21st Century Cures Act in the United States in 2021, patients have had the right to access their medical test results immediately, often before their physicians have had a chance to review them.

Lior Jankelson, NYU Grossman School of Medicine, New York, NY, and colleagues examined whether ChatGPT, a generative AI that has shown effectiveness in generating echocardiogram reports, can help clinicians efficiently explain echocardiogram reports to patients.

For this purpose, the researchers utilized a HIPAA-compliant ChatGPT (OpenAI) model to review and rewrite reports for 100 transthoracic echocardiograms from their institution. The median patient age was 66, with 23% presenting LV systolic dysfunction.

The AI-generated summaries were typically shorter than the full echocardiogram reports, with a median length of 1,216 characters compared to 2,150 characters. Five cardiologists assessed the rewrites, finding that 29% could be accepted without edits, 44% were acceptable with some revisions, 13% were neutral, and 14% were deemed unacceptable. Regarding accuracy, 84% of the rewrites were classified as “all true,” while 16% were considered “mostly correct.”

Of the 16 AI-generated rewrites with inaccuracies, eight were deemed "potentially dangerous," four required correction but were not considered dangerous, and four had errors that were unlikely to need correction.

Regarding the relevance of the rewrites, cardiologists found that 76% included "all of the important information," 15% included "most," 7% had "about half," and 2% contained "less than half." None of the rewrites with missing information were rated as "potentially dangerous." Nine required corrections but posed no danger, one had an "indeterminate need for correction," six were unlikely to need correction, and 12 were deemed "insignificant."

In terms of representing quantitative information, cardiologists strongly agreed with 54% of the rewrites, agreed with 36%, were neutral about 2%, and disagreed with 1%.

Twelve nonclinical reviewers assessed the rewrites for understandability. Compared to the original reports, 70% found the AI rewrites "much more" understandable, 27% "a little more," and 3% "equally" understandable. While 15% and 35% felt the rewrites would "strongly" and "slightly" reduce their worry, respectively, half were neutral or felt more anxious about the patient-oriented reports. However, 85% of nonclinical reviewers preferred having the AI-generated rewrites alongside the original reports. Inter-rater reliability was low among echocardiographers and only fair among nonclinical reviewers.

"Low inter-rater reliability of the study highlights that physicians vary significantly in their ECG report styles. This variation underscores the potential benefit of developing AI systems that could integrate individual practice patterns and preferences into the generated reports," the researchers concluded.

Reference:

Martin JA, Saric M, Vainrib AF, et al. Evaluating patient-oriented echocardiogram reports augmented by artificial intelligence. JACC Imaging. 2024;Epub ahead of print.