Evaluating AI Systems for Diabetic Retinopathy Screening

Introduction

Diabetes mellitus is a global health crisis affecting 1 in 10 adults. With rates quadrupling over the past two decades, approximately 537 million people are currently affected. This figure is projected to rise to 784 million by 2045. The financial burden of diabetes is substantial, with most costs directed toward managing complications. Diabetic retinopathy (DR) remains a leading cause of blindness among working-age individuals. Early detection through annual screenings and subsequent treatments can prevent or delay sight loss.

The English National Health Service (NHS) Diabetic Eye Screening Programme (DESP) faces challenges due to rising diabetes prevalence. Manual grading of approximately 13 million images each year is required. Automated retinal image analysis systems (ARIAS), which often utilise artificial intelligence (AI), offer an alternative by detecting those at medium to high risk of developing sight-threatening DR. This technology significantly expands grading capacity. However, vendor-led studies often present overly optimistic estimates compared to independent evaluations. Therefore, comparisons between multiple ARIAS using consistent datasets and computational environments are essential. Algorithmic fairness across diverse population subgroups must also be assessed before deployment.

Challenges in ARIAS Evaluation

Evaluating ARIAS presents several challenges. Vendor-independent comparisons on large, real-life population data are necessary for impartial evaluations. The largest, most ethnically diverse dataset from the North East London NHS DESP serves as a sustainable platform for independent evaluation of state-of-the-art ARIAS. This includes AI systems licensed as medical devices, such as those with FDA or CE Class IIa certification.

The evaluation methodology involves sample size calculations for equitable precision across population subgroups. This includes ethnicity, age, and levels of deprivation, along with a spectrum of diabetic eye disease. The platform can provide updated information on ARIAS performance at scale, offering feedback to vendors for algorithm improvements. Building transparent relationships with vendors is crucial for open-label publishing of ARIAS performance metrics.

Methodology and Findings

This methodology involves a comprehensive statistical analysis plan that ensures robustness and transparency. The evaluation faced challenges, including unpredictable timelines for ARIAS software delivery and bug fixes. The absence of a standard API for ARIAS and the intricate setup of the local Trusted Research Environment (TRE) introduced substantial time overheads. Some ARIAS vendors struggled to adapt their cloud-based systems to run offline, a requirement of NHS data governance standards. Furthermore, some vendors indicated their systems could only process high-resolution images, although the evaluation dataset contained lower resolution images that did not affect processing.

Conducting evaluations in a cloud-hosted TRE could have avoided many functional issues, allowing vendors to develop and test on a readily accessible platform. This would provide the research team with fast and flexible remote access, facilitating nearly real-world testing before deployment.

Evaluating AI Systems for Diabetic Retinopathy Screening

This methodological approach is suitable for evaluating AI in other healthcare imaging domains. Governmental, NHS, and healthcare provider stakeholders can employ this equitable methodology before implementation. The approach reflects findings from a recent governmental review on equity in medical devices, which highlighted the importance of testing AI medical devices in real-world contexts.

Key features of this study support the generalisability of findings, including multiple vendor participation and a large, diverse, clinically relevant dataset. A vendor-neutral research team executed the study using the same computational environment, independent of commercial interests. This encourages investment in health service provision, fostering trust in technological innovation.

Conclusion

Evaluating AI systems for diabetic retinopathy screening requires a comprehensive, transparent, and equitable approach. The methodology described here, using a large, diverse dataset and a vendor-neutral platform, ensures robust and impartial evaluations. Future evaluations should consider the comparative cost-effectiveness of ARIAS approaches and the benefits of cloud-hosted TREs. This approach not only enhances the accuracy of AI systems but also builds trust in their deployment across diverse healthcare settings.