Evaluating AI Systems for Diabetic Retinopathy Screening

By Staff Writer

August 15, 2024

Introduction

Diabetes mellitus is a global health crisis affecting 1 in 10 adults. With rates quadrupling over the past two decades, approximately 537 million people are currently affected. This figure is projected to rise to 784 million by 2045. The financial burden of diabetes is substantial, with most costs directed toward managing complications. Diabetic retinopathy (DR) remains a leading cause of blindness among working-age individuals. Early detection through annual screenings and subsequent treatments can prevent or delay sight loss.

The English National Health Service (NHS) Diabetic Eye Screening Programme (DESP) faces challenges due to rising diabetes prevalence. Manual grading of approximately 13 million images each year is required. Automated retinal image analysis systems (ARIAS), which often utilise artificial intelligence (AI), offer an alternative by detecting those at medium to high risk of developing sight-threatening DR. This technology significantly expands grading capacity. However, vendor-led studies often present overly optimistic estimates compared to independent evaluations. Therefore, comparisons between multiple ARIAS using consistent datasets and computational environments are essential. Algorithmic fairness across diverse population subgroups must also be assessed before deployment.

Challenges in ARIAS Evaluation

Evaluating ARIAS presents several challenges. Vendor-independent comparisons on large, real-life population data are necessary for impartial evaluations. The largest, most ethnically diverse dataset from the North East London NHS DESP serves as a sustainable platform for independent evaluation of state-of-the-art ARIAS. This includes AI systems licensed as medical devices, such as those with FDA or CE Class IIa certification.

The evaluation methodology involves sample size calculations for equitable precision across population subgroups. This includes ethnicity, age, and levels of deprivation, along with a spectrum of diabetic eye disease. The platform can provide updated information on ARIAS performance at scale, offering feedback to vendors for algorithm improvements. Building transparent relationships with vendors is crucial for open-label publishing of ARIAS performance metrics.

Methodology and Findings

This methodology involves a comprehensive statistical analysis plan that ensures robustness and transparency. The evaluation faced challenges, including unpredictable timelines for ARIAS software delivery and bug fixes. The absence of a standard API for ARIAS and the intricate setup of the local Trusted Research Environment (TRE) introduced substantial time overheads. Some ARIAS vendors struggled to adapt their cloud-based systems to run offline, a requirement of NHS data governance standards. Furthermore, some vendors indicated their systems could only process high-resolution images, although the evaluation dataset contained lower resolution images that did not affect processing.

Conducting evaluations in a cloud-hosted TRE could have avoided many functional issues, allowing vendors to develop and test on a readily accessible platform. This would provide the research team with fast and flexible remote access, facilitating nearly real-world testing before deployment.

Figure 1. Timelines for Vendor Enrollment, Software Preparation, and Verification

Evaluating AI Systems for Diabetic Retinopathy Screening

This methodological approach is suitable for evaluating AI in other healthcare imaging domains. Governmental, NHS, and healthcare provider stakeholders can employ this equitable methodology before implementation. The approach reflects findings from a recent governmental review on equity in medical devices, which highlighted the importance of testing AI medical devices in real-world contexts.

Key features of this study support the generalisability of findings, including multiple vendor participation and a large, diverse, clinically relevant dataset. A vendor-neutral research team executed the study using the same computational environment, independent of commercial interests. This encourages investment in health service provision, fostering trust in technological innovation.

Conclusion

Evaluating AI systems for diabetic retinopathy screening requires a comprehensive, transparent, and equitable approach. The methodology described here, using a large, diverse dataset and a vendor-neutral platform, ensures robust and impartial evaluations. Future evaluations should consider the comparative cost-effectiveness of ARIAS approaches and the benefits of cloud-hosted TREs. This approach not only enhances the accuracy of AI systems but also builds trust in their deployment across diverse healthcare settings.

Reference url

Recent Posts

oral health Africa
    

Oral Health in Africa: Promoting Collaborative Solutions

🦷 Is oral health taking a back seat in public health discussions in Africa?

A new article reveals alarming statistics about the high prevalence of untreated dental diseases across the continent and a critical shortage of oral health professionals. It emphasizes the urgent need for collaborative action among healthcare providers and policy-makers to integrate oral health into broader public health frameworks.

Discover how strengthening partnerships can pave the way for improved health outcomes and resource allocation in oral health.

#SyenzaNews #globalhealth #HealthEconomics

tislelizumab NSCLC treatment
        

Early Benefit Assessment of Tislelizumab NSCLC Treatment: Insights and Implications

🧐 How is the evolving treatment landscape for NSCLC affecting patient access to tislelizumab?

The German Federal Joint Committee (G-BA) has launched an early benefit assessment for tislelizumab as a second-line treatment for adults with advanced NSCLC. This assessment notably focuses on PD-L1 negative patients and highlights the need for additional data to substantiate its value amidst a shifting emphasis on first-line immunotherapy.

Explore the nuances of this assessment and its implications for future research and market access in the full article.

#SyenzaNews #oncology #MarketAccess

colorectal cancer screening
    

Advances in Colorectal Cancer Screening: Access and Cost

🚀 Is blood-based screening redefining colorectal cancer detection?

The Shield blood test offers a non-invasive alternative to colonoscopy—boosting screening uptake, but raising questions around effectiveness and value.

🔍 Discover how this innovation could reshape patient care, payer strategy, and health system costs.

#SyenzaNews #HealthcareInnovation #CostEffectiveness #DigitalTransformation

When you partner with Syenza, it’s like a Nuclear Fusion.

Our expertise are combined with yours, and we contribute clinical expertise and advanced degrees in health policy, health economics, systems analysis, public finance, business, and project management. You’ll also feel our high-impact global and local perspectives with cultural intelligence.

SPEAK WITH US

CORRESPONDENCE ADDRESS

1950 W. Corporate Way, Suite 95478
Anaheim, CA 92801, USA

© 2025 Syenza™. All rights reserved.