Evaluating AI Systems for Diabetic Retinopathy Screening

By Staff Writer

August 15, 2024

Introduction

Diabetes mellitus is a global health crisis affecting 1 in 10 adults. With rates quadrupling over the past two decades, approximately 537 million people are currently affected. This figure is projected to rise to 784 million by 2045. The financial burden of diabetes is substantial, with most costs directed toward managing complications. Diabetic retinopathy (DR) remains a leading cause of blindness among working-age individuals. Early detection through annual screenings and subsequent treatments can prevent or delay sight loss.

The English National Health Service (NHS) Diabetic Eye Screening Programme (DESP) faces challenges due to rising diabetes prevalence. Manual grading of approximately 13 million images each year is required. Automated retinal image analysis systems (ARIAS), which often utilise artificial intelligence (AI), offer an alternative by detecting those at medium to high risk of developing sight-threatening DR. This technology significantly expands grading capacity. However, vendor-led studies often present overly optimistic estimates compared to independent evaluations. Therefore, comparisons between multiple ARIAS using consistent datasets and computational environments are essential. Algorithmic fairness across diverse population subgroups must also be assessed before deployment.

Challenges in ARIAS Evaluation

Evaluating ARIAS presents several challenges. Vendor-independent comparisons on large, real-life population data are necessary for impartial evaluations. The largest, most ethnically diverse dataset from the North East London NHS DESP serves as a sustainable platform for independent evaluation of state-of-the-art ARIAS. This includes AI systems licensed as medical devices, such as those with FDA or CE Class IIa certification.

The evaluation methodology involves sample size calculations for equitable precision across population subgroups. This includes ethnicity, age, and levels of deprivation, along with a spectrum of diabetic eye disease. The platform can provide updated information on ARIAS performance at scale, offering feedback to vendors for algorithm improvements. Building transparent relationships with vendors is crucial for open-label publishing of ARIAS performance metrics.

Methodology and Findings

This methodology involves a comprehensive statistical analysis plan that ensures robustness and transparency. The evaluation faced challenges, including unpredictable timelines for ARIAS software delivery and bug fixes. The absence of a standard API for ARIAS and the intricate setup of the local Trusted Research Environment (TRE) introduced substantial time overheads. Some ARIAS vendors struggled to adapt their cloud-based systems to run offline, a requirement of NHS data governance standards. Furthermore, some vendors indicated their systems could only process high-resolution images, although the evaluation dataset contained lower resolution images that did not affect processing.

Conducting evaluations in a cloud-hosted TRE could have avoided many functional issues, allowing vendors to develop and test on a readily accessible platform. This would provide the research team with fast and flexible remote access, facilitating nearly real-world testing before deployment.

Figure 1. Timelines for Vendor Enrollment, Software Preparation, and Verification

Evaluating AI Systems for Diabetic Retinopathy Screening

This methodological approach is suitable for evaluating AI in other healthcare imaging domains. Governmental, NHS, and healthcare provider stakeholders can employ this equitable methodology before implementation. The approach reflects findings from a recent governmental review on equity in medical devices, which highlighted the importance of testing AI medical devices in real-world contexts.

Key features of this study support the generalisability of findings, including multiple vendor participation and a large, diverse, clinically relevant dataset. A vendor-neutral research team executed the study using the same computational environment, independent of commercial interests. This encourages investment in health service provision, fostering trust in technological innovation.

Conclusion

Evaluating AI systems for diabetic retinopathy screening requires a comprehensive, transparent, and equitable approach. The methodology described here, using a large, diverse dataset and a vendor-neutral platform, ensures robust and impartial evaluations. Future evaluations should consider the comparative cost-effectiveness of ARIAS approaches and the benefits of cloud-hosted TREs. This approach not only enhances the accuracy of AI systems but also builds trust in their deployment across diverse healthcare settings.

Reference url

Recent Posts

masitinib prostate cancer trial
        

Masitinib Prostate Cancer Trial Gains FDA and EMA Approval for Phase 3 Study

🚀 What’s the next big thing in prostate cancer treatment?

The recent FDA and EMA authorizations for the masitinib prostate cancer trial could revolutionize care for metastatic castrate-resistant prostate cancer (mCRPC). This groundbreaking study employs a biomarker-driven strategy to optimize treatment outcomes for patients like never before!

Dive into the full article to explore the implications of this targeted therapy and how it may change the future of precision medicine in oncology.

#SyenzaNews #Oncology #PrecisionMedicine #HealthTech

BEBT-908 DLBCL treatment
        

BEBT-908 DLBCL Treatment: A Milestone Approval for a First-in-Class Dual Inhibitor in China

🚀 Are we on the brink of a revolution in DLBCL treatment?

The recent conditional approval of **BEBT-908** by China’s National Medical Products Administration is not just a milestone for oncology, but potentially a game-changer for adults battling relapsed or refractory diffuse large B-cell lymphoma. With its dual-target mechanism, this first-in-class therapy offers promising efficacy and a well-thought-out access strategy that could reshape treatment standards.

Dive into the full article to discover how BEBT-908 is setting new benchmarks in both clinical outcomes and healthcare affordability.

#SyenzaNews #oncology #pharmaceuticals #MarketAccess

AI Cost Implications
    

AI Clinician in Pediatric ICU: A Review and HEOR Perspective

🤖 What could AI Clinician support mean for pediatric ICU care?

A new collaboration between Imperial College London and CHOC is developing an AI system to guide treatment decisions in PICUs. This article reviews the initiative and examines its broader health system implications—offering a Health Economics and Outcomes Research (HEOR) perspective on how AI may shape efficiency, equity, and value in critical care.

Read on to explore how clinical innovation intersects with healthcare economics.

#SyenzaNews #AIinHealthcare #HEOR #PediatricCare #HealthInnovation #PICU

When you partner with Syenza, it’s like a Nuclear Fusion.

Our expertise are combined with yours, and we contribute clinical expertise and advanced degrees in health policy, health economics, systems analysis, public finance, business, and project management. You’ll also feel our high-impact global and local perspectives with cultural intelligence.

SPEAK WITH US

CORRESPONDENCE ADDRESS

1950 W. Corporate Way, Suite 95478
Anaheim, CA 92801, USA

JOIN NEWSLETTER

© 2025 Syenza™. All rights reserved.