Evaluation of a Computer-Based System for Plus Disease Diagnosis in Retinopathy of Prematurity

Susan Koreen, Rony Gelman, M. Elena Martinez-Perez, Lei Jiang, Audina Berrocal, Ditte J. Hess, John T. Flynn, Michael F. Chiang

Research output: Contribution to journalArticle

62 Citations (Scopus)

Abstract

Objective: To measure accuracy and reliability of the computer-based Retinal Image Multiscale Analysis (RISA) system compared with those of recognized retinopathy of prematurity (ROP) experts, for plus disease diagnosis. Design: Evaluation of diagnostic test or technology. Participants: Eleven recognized ROP experts and the RISA image analysis system interpreted a set of 20 wide-angle retinal photographs for presence of plus disease. Methods: All experts used a secure Web site to review independently 20 images for presence of plus disease. Images were also analyzed by measuring individual computer-based system parameters (integrated curvature [IC], diameter, and tortuosity index) for arterioles and venules and by computing linear combinations and logical combinations of those parameters. Performance was compared with a reference standard, defined as the majority vote of experts. Main Outcome Measures: Diagnostic accuracy was measured by calculating sensitivity, specificity, and receiver operating characteristic area under the curve (AUC) for plus disease diagnosis by each expert, and by each computer-based system parameter, compared with the reference standard. Diagnostic agreement was measured by calculating the mean κ value of each expert compared with all other experts and the mean κ value of each computer-based system parameter compared with all experts. Results: Among the 11 experts, sensitivity ranged from 0.167 to 1.000, specificity ranged from 0.714 to 1.000, AUC ranged from 0.798 to 1.000, and mean κ compared with all other experts ranged from 0.288 to 0.689. Among individual computer system parameters, arteriolar IC had the highest diagnostic accuracy, with sensitivity of 1.000; specificity, 0.846; and AUC, 0.962. Arteriolar IC had the highest diagnostic agreement with experts, with a mean κ value of 0.578. Conclusions: A computer-based image analysis system has the potential to perform comparably to recognized ROP experts for plus disease diagnosis.

Original languageEnglish
JournalOphthalmology
Volume114
Issue number12
DOIs
StatePublished - Dec 1 2007

Fingerprint

Retinopathy of Prematurity
Computer Systems
Area Under Curve
Sensitivity and Specificity
Venules
Arterioles
Routine Diagnostic Tests
ROC Curve
Outcome Assessment (Health Care)
Technology

ASJC Scopus subject areas

  • Ophthalmology

Cite this

Koreen, S., Gelman, R., Martinez-Perez, M. E., Jiang, L., Berrocal, A., Hess, D. J., ... Chiang, M. F. (2007). Evaluation of a Computer-Based System for Plus Disease Diagnosis in Retinopathy of Prematurity. Ophthalmology, 114(12). https://doi.org/10.1016/j.ophtha.2007.10.006

Evaluation of a Computer-Based System for Plus Disease Diagnosis in Retinopathy of Prematurity. / Koreen, Susan; Gelman, Rony; Martinez-Perez, M. Elena; Jiang, Lei; Berrocal, Audina; Hess, Ditte J.; Flynn, John T.; Chiang, Michael F.

In: Ophthalmology, Vol. 114, No. 12, 01.12.2007.

Research output: Contribution to journalArticle

Koreen, S, Gelman, R, Martinez-Perez, ME, Jiang, L, Berrocal, A, Hess, DJ, Flynn, JT & Chiang, MF 2007, 'Evaluation of a Computer-Based System for Plus Disease Diagnosis in Retinopathy of Prematurity', Ophthalmology, vol. 114, no. 12. https://doi.org/10.1016/j.ophtha.2007.10.006
Koreen, Susan ; Gelman, Rony ; Martinez-Perez, M. Elena ; Jiang, Lei ; Berrocal, Audina ; Hess, Ditte J. ; Flynn, John T. ; Chiang, Michael F. / Evaluation of a Computer-Based System for Plus Disease Diagnosis in Retinopathy of Prematurity. In: Ophthalmology. 2007 ; Vol. 114, No. 12.
@article{1da3274b8e334bb5aab8bf8e734d9747,
title = "Evaluation of a Computer-Based System for Plus Disease Diagnosis in Retinopathy of Prematurity",
abstract = "Objective: To measure accuracy and reliability of the computer-based Retinal Image Multiscale Analysis (RISA) system compared with those of recognized retinopathy of prematurity (ROP) experts, for plus disease diagnosis. Design: Evaluation of diagnostic test or technology. Participants: Eleven recognized ROP experts and the RISA image analysis system interpreted a set of 20 wide-angle retinal photographs for presence of plus disease. Methods: All experts used a secure Web site to review independently 20 images for presence of plus disease. Images were also analyzed by measuring individual computer-based system parameters (integrated curvature [IC], diameter, and tortuosity index) for arterioles and venules and by computing linear combinations and logical combinations of those parameters. Performance was compared with a reference standard, defined as the majority vote of experts. Main Outcome Measures: Diagnostic accuracy was measured by calculating sensitivity, specificity, and receiver operating characteristic area under the curve (AUC) for plus disease diagnosis by each expert, and by each computer-based system parameter, compared with the reference standard. Diagnostic agreement was measured by calculating the mean κ value of each expert compared with all other experts and the mean κ value of each computer-based system parameter compared with all experts. Results: Among the 11 experts, sensitivity ranged from 0.167 to 1.000, specificity ranged from 0.714 to 1.000, AUC ranged from 0.798 to 1.000, and mean κ compared with all other experts ranged from 0.288 to 0.689. Among individual computer system parameters, arteriolar IC had the highest diagnostic accuracy, with sensitivity of 1.000; specificity, 0.846; and AUC, 0.962. Arteriolar IC had the highest diagnostic agreement with experts, with a mean κ value of 0.578. Conclusions: A computer-based image analysis system has the potential to perform comparably to recognized ROP experts for plus disease diagnosis.",
author = "Susan Koreen and Rony Gelman and Martinez-Perez, {M. Elena} and Lei Jiang and Audina Berrocal and Hess, {Ditte J.} and Flynn, {John T.} and Chiang, {Michael F.}",
year = "2007",
month = "12",
day = "1",
doi = "10.1016/j.ophtha.2007.10.006",
language = "English",
volume = "114",
journal = "Ophthalmology",
issn = "0161-6420",
publisher = "Elsevier Inc.",
number = "12",

}

TY - JOUR

T1 - Evaluation of a Computer-Based System for Plus Disease Diagnosis in Retinopathy of Prematurity

AU - Koreen, Susan

AU - Gelman, Rony

AU - Martinez-Perez, M. Elena

AU - Jiang, Lei

AU - Berrocal, Audina

AU - Hess, Ditte J.

AU - Flynn, John T.

AU - Chiang, Michael F.

PY - 2007/12/1

Y1 - 2007/12/1

N2 - Objective: To measure accuracy and reliability of the computer-based Retinal Image Multiscale Analysis (RISA) system compared with those of recognized retinopathy of prematurity (ROP) experts, for plus disease diagnosis. Design: Evaluation of diagnostic test or technology. Participants: Eleven recognized ROP experts and the RISA image analysis system interpreted a set of 20 wide-angle retinal photographs for presence of plus disease. Methods: All experts used a secure Web site to review independently 20 images for presence of plus disease. Images were also analyzed by measuring individual computer-based system parameters (integrated curvature [IC], diameter, and tortuosity index) for arterioles and venules and by computing linear combinations and logical combinations of those parameters. Performance was compared with a reference standard, defined as the majority vote of experts. Main Outcome Measures: Diagnostic accuracy was measured by calculating sensitivity, specificity, and receiver operating characteristic area under the curve (AUC) for plus disease diagnosis by each expert, and by each computer-based system parameter, compared with the reference standard. Diagnostic agreement was measured by calculating the mean κ value of each expert compared with all other experts and the mean κ value of each computer-based system parameter compared with all experts. Results: Among the 11 experts, sensitivity ranged from 0.167 to 1.000, specificity ranged from 0.714 to 1.000, AUC ranged from 0.798 to 1.000, and mean κ compared with all other experts ranged from 0.288 to 0.689. Among individual computer system parameters, arteriolar IC had the highest diagnostic accuracy, with sensitivity of 1.000; specificity, 0.846; and AUC, 0.962. Arteriolar IC had the highest diagnostic agreement with experts, with a mean κ value of 0.578. Conclusions: A computer-based image analysis system has the potential to perform comparably to recognized ROP experts for plus disease diagnosis.

AB - Objective: To measure accuracy and reliability of the computer-based Retinal Image Multiscale Analysis (RISA) system compared with those of recognized retinopathy of prematurity (ROP) experts, for plus disease diagnosis. Design: Evaluation of diagnostic test or technology. Participants: Eleven recognized ROP experts and the RISA image analysis system interpreted a set of 20 wide-angle retinal photographs for presence of plus disease. Methods: All experts used a secure Web site to review independently 20 images for presence of plus disease. Images were also analyzed by measuring individual computer-based system parameters (integrated curvature [IC], diameter, and tortuosity index) for arterioles and venules and by computing linear combinations and logical combinations of those parameters. Performance was compared with a reference standard, defined as the majority vote of experts. Main Outcome Measures: Diagnostic accuracy was measured by calculating sensitivity, specificity, and receiver operating characteristic area under the curve (AUC) for plus disease diagnosis by each expert, and by each computer-based system parameter, compared with the reference standard. Diagnostic agreement was measured by calculating the mean κ value of each expert compared with all other experts and the mean κ value of each computer-based system parameter compared with all experts. Results: Among the 11 experts, sensitivity ranged from 0.167 to 1.000, specificity ranged from 0.714 to 1.000, AUC ranged from 0.798 to 1.000, and mean κ compared with all other experts ranged from 0.288 to 0.689. Among individual computer system parameters, arteriolar IC had the highest diagnostic accuracy, with sensitivity of 1.000; specificity, 0.846; and AUC, 0.962. Arteriolar IC had the highest diagnostic agreement with experts, with a mean κ value of 0.578. Conclusions: A computer-based image analysis system has the potential to perform comparably to recognized ROP experts for plus disease diagnosis.

UR - http://www.scopus.com/inward/record.url?scp=36549002810&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=36549002810&partnerID=8YFLogxK

U2 - 10.1016/j.ophtha.2007.10.006

DO - 10.1016/j.ophtha.2007.10.006

M3 - Article

C2 - 18054630

AN - SCOPUS:36549002810

VL - 114

JO - Ophthalmology

JF - Ophthalmology

SN - 0161-6420

IS - 12

ER -