Visual Quality Control of Assessment of AI-Assisted High-Volume CMR Segmentation in The UK Biobank

Abstract

Background Automated algorithms are being used regularly to analyse cardiac magnetic resonance (CMR) images. Validating data output reliability from these methods is necessary to enable widespread adoption. We outline a visual quality control (QC) process for image analysis performed using automated batch processing methods. We aim to report the performance of automated methods and the reliability of replacing visual checks with a statistical outlier removal approach in UK Biobank CMR scans. Methods CMR scans included (n=1987) were from the UK Biobank COVID imaging study. Automated batch processing software developed by Circle Cardiovascular Imaging Inc (CVI 42) was used to extract chamber volumetric data, strain, native T1 and aortic flow data. The video outputs of the automated image analysis (~ 62,000 videos and 2000 images) were visually reviewed and rated by six experienced clinicians using a custom-built R Shiny app. The standardised approach (consisting of grading 1,2,3 for good, satisfactory or poor quality respectively) was agreed during two rounds of scoring followed by open discussion. Interobserver variability was assessed using Gwet's second order agreement co-efficient (AC2) analysis. The data output from scans passing visual QC was compared with data from a statistical outlier removal QC method, using t-test analysis, in a subset of healthy individuals from baseline imaging (n = 1069). Results The quality of the automated image analysis was very high with >95% of scans passing the visual QC (scored 1 or 2) for all modalities of image analysis. There was good interobserver agreement with overall AC2 of 0.91(+/- 0.14, 95% confidence interval (0.84,0.94)). There was no difference in the overall distribution of data and derived average values from visual QC process or statistical outlier removal in a subset of healthy individuals from this study. Conclusion The quality of automated image analysis is very high using the prototypes developed by CVI42 for the UK Biobank imaging study CMR scans. Therefore, larger UK Biobank datasets analysed using these automated algorithms do not need in-depth visual QC. Statistical outlier removal is a sufficient QC measure, with operator discretion for visual checks based on their respective population or research aim.