Measuring and Teaching Confidence Calibration Among Radiologists: A Multi-Institution Study

Michael L. Richardson, Behrang Amini, Nicholas M. Beckmann, Ty K. Subhawong

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Objective: Our purpose was to assess the calibration of resident, fellow, and attending radiologists on a simple image classification task (presence or absence of an anterior cruciate ligament [ACL] tear based on interpretation of sagittal proton density, fat-saturated MR images) and to assess whether teaching residents could improve their calibration. Methods: We created a test containing 30 randomized, sagittal, proton density, fat-saturated MR images of the ACL (15 normal, 15 torn). This test was administered in person to 20 trainees and 3 attendings at one medical center in one state. An online version of the test was given to 23 trainees and 14 attendings from 11 other medical centers in nine other states. Subjects were asked to give their confidence level (0%-100%) that each ACL was torn. Results: Cross-sectional data were collected from 60 radiologists (mean time after medical school = 9.3 years, minimum = 1 year, maximum = 36 years). This demonstrated a statistically significant improvement in calibration as a function of increasing experience (P = .020). Longitudinal data were collected from 12 trainees at the start and end of their musculoskeletal radiology rotation, with an intervening review of the primary and secondary signs of ACL tear on MR. A statistically significant improvement in calibration was noted during the rotation (P = .028). Conclusions: Confidence calibration is a promising tool for quality improvement and radiologist self-assessment. Our study showed that calibration loss improves with experience in radiologists tested on a common and clinically important image classification task. We also demonstrated that calibration can be successfully taught to residents over a relatively short period (2-4 weeks).

Original languageEnglish (US)
Pages (from-to)1314-1321
Number of pages8
JournalJournal of the American College of Radiology
Volume17
Issue number10
DOIs
StatePublished - Oct 2020

Keywords

  • Brier score
  • confidence calibration
  • quality improvement
  • radiologist self-assessment
  • radiology resident education

ASJC Scopus subject areas

  • Radiology Nuclear Medicine and imaging

Fingerprint Dive into the research topics of 'Measuring and Teaching Confidence Calibration Among Radiologists: A Multi-Institution Study'. Together they form a unique fingerprint.

Cite this