Rationale and Objectives. In practice readers must often choose between multiple diagnoses. For assessing reader accuracy in these settings, Obuchowski et al have proposed the "differential diagnosis" method, which derives all pairwise estimates of accuracy for the various diagnoses, along with summary measures of accuracy. The current study assessed the correspondence between the differential diagnosis method and conventional binary-truth state experiments. Materials and Methods. Two empirical studies were conducted at two institutions with different readers and diagnostic tests. Readers used the differential diagnosis format to interpret a set of cases. In subsequent readings they interpreted the cases in binary-truth state experiments. Spearman rank correlation coefficients and the percentages of agreement in scores were computed, and the areas under the receiver operating characteristic curves were estimated and compared. Results. The between-format Spearman rank correlation coefficients were 0.697-0.718 and 0.750-0.780 for the two studies; the between-reader correlations were 0.417 and 0.792, respectively. The percentages of agreement between formats for the two studies were 50.0%-51.7% and 72.9%-78.8%; the percentages of agreement between readers were 45.0% and 80%, respectively. In the first study there were several significant differences in the areas under receiver operating characteristic curves; in the second study these differences were small. Conclusion. The differences observed between the two formats can be attributed to within-reader variability and inherent differences in the questions posed to readers in the multiple-diagnoses versus binary-truth state reading sessions. The differential diagnosis format is useful for estimating accuracy when there are multiple possible diagnoses.
- Diagnostic radiology
- Observer performance
- Receiver operating characteristic curve (ROC)
ASJC Scopus subject areas
- Radiology Nuclear Medicine and imaging