Evaluating AI Systems for Diabetic Retinopathy Screening

By Staff Writer

August 15, 2024

Introduction

Diabetes mellitus is a global health crisis affecting 1 in 10 adults. With rates quadrupling over the past two decades, approximately 537 million people are currently affected. This figure is projected to rise to 784 million by 2045. The financial burden of diabetes is substantial, with most costs directed toward managing complications. Diabetic retinopathy (DR) remains a leading cause of blindness among working-age individuals. Early detection through annual screenings and subsequent treatments can prevent or delay sight loss.

The English National Health Service (NHS) Diabetic Eye Screening Programme (DESP) faces challenges due to rising diabetes prevalence. Manual grading of approximately 13 million images each year is required. Automated retinal image analysis systems (ARIAS), which often utilise artificial intelligence (AI), offer an alternative by detecting those at medium to high risk of developing sight-threatening DR. This technology significantly expands grading capacity. However, vendor-led studies often present overly optimistic estimates compared to independent evaluations. Therefore, comparisons between multiple ARIAS using consistent datasets and computational environments are essential. Algorithmic fairness across diverse population subgroups must also be assessed before deployment.

Challenges in ARIAS Evaluation

Evaluating ARIAS presents several challenges. Vendor-independent comparisons on large, real-life population data are necessary for impartial evaluations. The largest, most ethnically diverse dataset from the North East London NHS DESP serves as a sustainable platform for independent evaluation of state-of-the-art ARIAS. This includes AI systems licensed as medical devices, such as those with FDA or CE Class IIa certification.

The evaluation methodology involves sample size calculations for equitable precision across population subgroups. This includes ethnicity, age, and levels of deprivation, along with a spectrum of diabetic eye disease. The platform can provide updated information on ARIAS performance at scale, offering feedback to vendors for algorithm improvements. Building transparent relationships with vendors is crucial for open-label publishing of ARIAS performance metrics.

Methodology and Findings

This methodology involves a comprehensive statistical analysis plan that ensures robustness and transparency. The evaluation faced challenges, including unpredictable timelines for ARIAS software delivery and bug fixes. The absence of a standard API for ARIAS and the intricate setup of the local Trusted Research Environment (TRE) introduced substantial time overheads. Some ARIAS vendors struggled to adapt their cloud-based systems to run offline, a requirement of NHS data governance standards. Furthermore, some vendors indicated their systems could only process high-resolution images, although the evaluation dataset contained lower resolution images that did not affect processing.

Conducting evaluations in a cloud-hosted TRE could have avoided many functional issues, allowing vendors to develop and test on a readily accessible platform. This would provide the research team with fast and flexible remote access, facilitating nearly real-world testing before deployment.

Figure 1. Timelines for Vendor Enrollment, Software Preparation, and Verification

Evaluating AI Systems for Diabetic Retinopathy Screening

This methodological approach is suitable for evaluating AI in other healthcare imaging domains. Governmental, NHS, and healthcare provider stakeholders can employ this equitable methodology before implementation. The approach reflects findings from a recent governmental review on equity in medical devices, which highlighted the importance of testing AI medical devices in real-world contexts.

Key features of this study support the generalisability of findings, including multiple vendor participation and a large, diverse, clinically relevant dataset. A vendor-neutral research team executed the study using the same computational environment, independent of commercial interests. This encourages investment in health service provision, fostering trust in technological innovation.

Conclusion

Evaluating AI systems for diabetic retinopathy screening requires a comprehensive, transparent, and equitable approach. The methodology described here, using a large, diverse dataset and a vendor-neutral platform, ensures robust and impartial evaluations. Future evaluations should consider the comparative cost-effectiveness of ARIAS approaches and the benefits of cloud-hosted TREs. This approach not only enhances the accuracy of AI systems but also builds trust in their deployment across diverse healthcare settings.

Reference url

Recent Posts

EFPIA Response to Tariffs: Concerns Over U.S.-EU Trade Agreement Impact on Pharma Industry

By Staff Writer

August 22, 2025

The European Federation of Pharmaceutical Industries and Associations (EFPIA) has expressed significant concern over the new U.S.-EU Joint Framework Agreement. It is particularly worried about potential tariffs of 15% on pharmaceutical products. The organization argues these tariffs break a three...
TAVI Cost-Effectiveness in Brazil: Balancing Coverage Expansion and Economic Viability

By Staff Writer

August 21, 2025

The focus on TAVI cost-effectiveness Brazil has gained attention. The country is assessing the expansion of coverage for transcatheter aortic valve implantation (TAVI). This would include patients with severe aortic stenosis who are classified as low surgical risk (STS-PROM <4%) as a first-lin...
Evolving Strategies in Type 2 Diabetes Management: NICE Guideline Update 2025
Type 2 diabetes management in adults is undergoing significant changes with the draft 2025 NICE guideline update. What are the main improvements in medication, monitoring, and patient care for adults with type 2 diabetes? The answer: The new NICE recommendations prioritize a broader use of SGL...