AI System Shows Consistent Accuracy in Echocardiogram Interpretation: JAMA
A recent study published in JAMA has demonstrated that an AI system can reliably interpret echocardiograms with high accuracy across different geographic locations and time periods, using both complete and limited data sets. The system shows potential for integration as an adjunct reader in echocardiography labs or as a screening tool in point-of-care settings, pending prospective evaluation within actual clinical workflows. The study was conducted by Gregory H. and colleagues.
The PanEcho AI model was trained on transthoracic echocardiographic (TTE) information from routine clinical practice at Yale New Haven Health System (YNHHS) from January 2016 to June 2022. The training dataset consisted of 1.2 million echocardiographic videos derived from 32,265 TTE examinations of 24,405 patients. Internal validation was performed with a temporally distinct YNHHS cohort during July-December 2022. Further, the model was externally validated in four different health care cohorts and openly released for wider assessment. The system was trained on a multitask deep learning model that could process both diagnostic classification and echocardiographic parameter estimation simultaneously.
The performance of PanEcho was assessed using two main metrics:
• Area Under the Receiver Operating Characteristic Curve (AUC): For diagnostic classification.
• Mean Absolute Error (MAE): For estimation of echocardiographic parameters.
Key Findings
Diagnostic Accuracy:
• PanEcho performed 18 diagnostic classification tasks with a median AUC of 0.91 (interquartile range [IQR]: 0.88–0.93) in internal validation.
Parameter Estimation:
• For 21 echocardiographic measurements, the median normalized mean absolute error was 0.13 (IQR: 0.10–0.18).
• Left ventricular ejection fraction (LVEF): Estimated with mean absolute error of 4.2% (internal) and 4.5% (external).
• Detection of moderate or greater left ventricular systolic dysfunction: AUC of 0.98 (internal) and 0.99 (external).
• Right ventricular systolic dysfunction: AUC of 0.93 (internal) and 0.94 (external).
• Severe aortic stenosis: AUC of 0.98 (internal) and 1.00 (external).
Performance in Limited Protocols:
• Abbreviated TTE cohort: 15 diagnosis tasks, median AUC of 0.91 (IQR: 0.87–0.94)
• Emergency department point-of-care cohort: 14 diagnosis tasks, median AUC of 0.85 (IQR: 0.77–0.87).
In conclusion, the present study showed that the AI model PanEcho was able to accurately interpret a large set of echocardiographic parameters and diagnostic labels at high rates in various real-world clinical environments. The results substantiate the future integration of PanEcho into standard cardiovascular imaging practice, especially as a tool to enhance efficiency and consistency in reporting echocardiograms.
Reference:
