Fundamentals of endoscopic surgery: Creation and validation of the hands-on test

Melina C. Vassiliou, Brian J. Dunkin, Gerald M. Fried, John D. Mellinger, Thadeus Trus, Pepa Kaneva, Calvin Lyons, James R. Korndorffer, Michael Ujiki, Vic Velanovich, Michael L. Kochman, Shawn Tsuda, Jose M Martinez, Daniel J. Scott, Gary Korus, Adrian Park, Jeffrey M. Marks

Research output: Contribution to journalArticle

51 Citations (Scopus)

Abstract

Background: The Fundamentals of Endoscopic Surgery™ (FES) program consists of online materials and didactic and skills-based tests. All components were designed to measure the skills and knowledge required to perform safe flexible endoscopy. The purpose of this multicenter study was to evaluate the reliability and validity of the hands-on component of the FES examination, and to establish the pass score. Methods: Expert endoscopists identified the critical skill set required for flexible endoscopy. They were then modeled in a virtual reality simulator (GI Mentor™ II, Simbionix™ Ltd., Airport City, Israel) to create five tasks and metrics. Scores were designed to measure both speed and precision. Validity evidence was assessed by correlating performance with self-reported endoscopic experience (surgeons and gastroenterologists [GIs]). Internal consistency of each test task was assessed using Cronbach's alpha. Test-retest reliability was determined by having the same participant perform the test a second time and comparing their scores. Passing scores were determined by a contrasting groups methodology and use of receiver operating characteristic curves. Results: A total of 160 participants (17 % GIs) performed the simulator test. Scores on the five tasks showed good internal consistency reliability and all had significant correlations with endoscopic experience. Total FES scores correlated 0.73, with participants' level of endoscopic experience providing evidence of their validity, and their internal consistency reliability (Cronbach's alpha) was 0.82. Test-retest reliability was assessed in 11 participants, and the intraclass correlation was 0.85. The passing score was determined and is estimated to have a sensitivity (true positive rate) of 0.81 and a 1-specificity (false positive rate) of 0.21. Conclusions: The FES hands-on skills test examines the basic procedural components required to perform safe flexible endoscopy. It meets rigorous standards of reliability and validity required for high-stakes examinations, and, together with the knowledge component, may help contribute to the definition and determination of competence in endoscopy.

Original languageEnglish
Pages (from-to)704-711
Number of pages8
JournalSurgical Endoscopy and Other Interventional Techniques
Volume28
Issue number3
DOIs
StatePublished - Jan 1 2014

Fingerprint

Reproducibility of Results
Endoscopy
Airports
Mentors
Israel
ROC Curve
Mental Competency
Multicenter Studies
Gastroenterologists

Keywords

  • Assessment
  • Education
  • Examination
  • FES
  • Flexible endoscopy
  • Training

ASJC Scopus subject areas

  • Surgery
  • Medicine(all)

Cite this

Vassiliou, M. C., Dunkin, B. J., Fried, G. M., Mellinger, J. D., Trus, T., Kaneva, P., ... Marks, J. M. (2014). Fundamentals of endoscopic surgery: Creation and validation of the hands-on test. Surgical Endoscopy and Other Interventional Techniques, 28(3), 704-711. https://doi.org/10.1007/s00464-013-3298-4

Fundamentals of endoscopic surgery : Creation and validation of the hands-on test. / Vassiliou, Melina C.; Dunkin, Brian J.; Fried, Gerald M.; Mellinger, John D.; Trus, Thadeus; Kaneva, Pepa; Lyons, Calvin; Korndorffer, James R.; Ujiki, Michael; Velanovich, Vic; Kochman, Michael L.; Tsuda, Shawn; Martinez, Jose M; Scott, Daniel J.; Korus, Gary; Park, Adrian; Marks, Jeffrey M.

In: Surgical Endoscopy and Other Interventional Techniques, Vol. 28, No. 3, 01.01.2014, p. 704-711.

Research output: Contribution to journalArticle

Vassiliou, MC, Dunkin, BJ, Fried, GM, Mellinger, JD, Trus, T, Kaneva, P, Lyons, C, Korndorffer, JR, Ujiki, M, Velanovich, V, Kochman, ML, Tsuda, S, Martinez, JM, Scott, DJ, Korus, G, Park, A & Marks, JM 2014, 'Fundamentals of endoscopic surgery: Creation and validation of the hands-on test', Surgical Endoscopy and Other Interventional Techniques, vol. 28, no. 3, pp. 704-711. https://doi.org/10.1007/s00464-013-3298-4
Vassiliou, Melina C. ; Dunkin, Brian J. ; Fried, Gerald M. ; Mellinger, John D. ; Trus, Thadeus ; Kaneva, Pepa ; Lyons, Calvin ; Korndorffer, James R. ; Ujiki, Michael ; Velanovich, Vic ; Kochman, Michael L. ; Tsuda, Shawn ; Martinez, Jose M ; Scott, Daniel J. ; Korus, Gary ; Park, Adrian ; Marks, Jeffrey M. / Fundamentals of endoscopic surgery : Creation and validation of the hands-on test. In: Surgical Endoscopy and Other Interventional Techniques. 2014 ; Vol. 28, No. 3. pp. 704-711.
@article{88d5356db6c44027b501dd7481430f5c,
title = "Fundamentals of endoscopic surgery: Creation and validation of the hands-on test",
abstract = "Background: The Fundamentals of Endoscopic Surgery™ (FES) program consists of online materials and didactic and skills-based tests. All components were designed to measure the skills and knowledge required to perform safe flexible endoscopy. The purpose of this multicenter study was to evaluate the reliability and validity of the hands-on component of the FES examination, and to establish the pass score. Methods: Expert endoscopists identified the critical skill set required for flexible endoscopy. They were then modeled in a virtual reality simulator (GI Mentor™ II, Simbionix™ Ltd., Airport City, Israel) to create five tasks and metrics. Scores were designed to measure both speed and precision. Validity evidence was assessed by correlating performance with self-reported endoscopic experience (surgeons and gastroenterologists [GIs]). Internal consistency of each test task was assessed using Cronbach's alpha. Test-retest reliability was determined by having the same participant perform the test a second time and comparing their scores. Passing scores were determined by a contrasting groups methodology and use of receiver operating characteristic curves. Results: A total of 160 participants (17 {\%} GIs) performed the simulator test. Scores on the five tasks showed good internal consistency reliability and all had significant correlations with endoscopic experience. Total FES scores correlated 0.73, with participants' level of endoscopic experience providing evidence of their validity, and their internal consistency reliability (Cronbach's alpha) was 0.82. Test-retest reliability was assessed in 11 participants, and the intraclass correlation was 0.85. The passing score was determined and is estimated to have a sensitivity (true positive rate) of 0.81 and a 1-specificity (false positive rate) of 0.21. Conclusions: The FES hands-on skills test examines the basic procedural components required to perform safe flexible endoscopy. It meets rigorous standards of reliability and validity required for high-stakes examinations, and, together with the knowledge component, may help contribute to the definition and determination of competence in endoscopy.",
keywords = "Assessment, Education, Examination, FES, Flexible endoscopy, Training",
author = "Vassiliou, {Melina C.} and Dunkin, {Brian J.} and Fried, {Gerald M.} and Mellinger, {John D.} and Thadeus Trus and Pepa Kaneva and Calvin Lyons and Korndorffer, {James R.} and Michael Ujiki and Vic Velanovich and Kochman, {Michael L.} and Shawn Tsuda and Martinez, {Jose M} and Scott, {Daniel J.} and Gary Korus and Adrian Park and Marks, {Jeffrey M.}",
year = "2014",
month = "1",
day = "1",
doi = "10.1007/s00464-013-3298-4",
language = "English",
volume = "28",
pages = "704--711",
journal = "Surgical Endoscopy and Other Interventional Techniques",
issn = "0930-2794",
publisher = "Springer New York",
number = "3",

}

TY - JOUR

T1 - Fundamentals of endoscopic surgery

T2 - Creation and validation of the hands-on test

AU - Vassiliou, Melina C.

AU - Dunkin, Brian J.

AU - Fried, Gerald M.

AU - Mellinger, John D.

AU - Trus, Thadeus

AU - Kaneva, Pepa

AU - Lyons, Calvin

AU - Korndorffer, James R.

AU - Ujiki, Michael

AU - Velanovich, Vic

AU - Kochman, Michael L.

AU - Tsuda, Shawn

AU - Martinez, Jose M

AU - Scott, Daniel J.

AU - Korus, Gary

AU - Park, Adrian

AU - Marks, Jeffrey M.

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Background: The Fundamentals of Endoscopic Surgery™ (FES) program consists of online materials and didactic and skills-based tests. All components were designed to measure the skills and knowledge required to perform safe flexible endoscopy. The purpose of this multicenter study was to evaluate the reliability and validity of the hands-on component of the FES examination, and to establish the pass score. Methods: Expert endoscopists identified the critical skill set required for flexible endoscopy. They were then modeled in a virtual reality simulator (GI Mentor™ II, Simbionix™ Ltd., Airport City, Israel) to create five tasks and metrics. Scores were designed to measure both speed and precision. Validity evidence was assessed by correlating performance with self-reported endoscopic experience (surgeons and gastroenterologists [GIs]). Internal consistency of each test task was assessed using Cronbach's alpha. Test-retest reliability was determined by having the same participant perform the test a second time and comparing their scores. Passing scores were determined by a contrasting groups methodology and use of receiver operating characteristic curves. Results: A total of 160 participants (17 % GIs) performed the simulator test. Scores on the five tasks showed good internal consistency reliability and all had significant correlations with endoscopic experience. Total FES scores correlated 0.73, with participants' level of endoscopic experience providing evidence of their validity, and their internal consistency reliability (Cronbach's alpha) was 0.82. Test-retest reliability was assessed in 11 participants, and the intraclass correlation was 0.85. The passing score was determined and is estimated to have a sensitivity (true positive rate) of 0.81 and a 1-specificity (false positive rate) of 0.21. Conclusions: The FES hands-on skills test examines the basic procedural components required to perform safe flexible endoscopy. It meets rigorous standards of reliability and validity required for high-stakes examinations, and, together with the knowledge component, may help contribute to the definition and determination of competence in endoscopy.

AB - Background: The Fundamentals of Endoscopic Surgery™ (FES) program consists of online materials and didactic and skills-based tests. All components were designed to measure the skills and knowledge required to perform safe flexible endoscopy. The purpose of this multicenter study was to evaluate the reliability and validity of the hands-on component of the FES examination, and to establish the pass score. Methods: Expert endoscopists identified the critical skill set required for flexible endoscopy. They were then modeled in a virtual reality simulator (GI Mentor™ II, Simbionix™ Ltd., Airport City, Israel) to create five tasks and metrics. Scores were designed to measure both speed and precision. Validity evidence was assessed by correlating performance with self-reported endoscopic experience (surgeons and gastroenterologists [GIs]). Internal consistency of each test task was assessed using Cronbach's alpha. Test-retest reliability was determined by having the same participant perform the test a second time and comparing their scores. Passing scores were determined by a contrasting groups methodology and use of receiver operating characteristic curves. Results: A total of 160 participants (17 % GIs) performed the simulator test. Scores on the five tasks showed good internal consistency reliability and all had significant correlations with endoscopic experience. Total FES scores correlated 0.73, with participants' level of endoscopic experience providing evidence of their validity, and their internal consistency reliability (Cronbach's alpha) was 0.82. Test-retest reliability was assessed in 11 participants, and the intraclass correlation was 0.85. The passing score was determined and is estimated to have a sensitivity (true positive rate) of 0.81 and a 1-specificity (false positive rate) of 0.21. Conclusions: The FES hands-on skills test examines the basic procedural components required to perform safe flexible endoscopy. It meets rigorous standards of reliability and validity required for high-stakes examinations, and, together with the knowledge component, may help contribute to the definition and determination of competence in endoscopy.

KW - Assessment

KW - Education

KW - Examination

KW - FES

KW - Flexible endoscopy

KW - Training

UR - http://www.scopus.com/inward/record.url?scp=84896101981&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84896101981&partnerID=8YFLogxK

U2 - 10.1007/s00464-013-3298-4

DO - 10.1007/s00464-013-3298-4

M3 - Article

C2 - 24253562

AN - SCOPUS:84896101981

VL - 28

SP - 704

EP - 711

JO - Surgical Endoscopy and Other Interventional Techniques

JF - Surgical Endoscopy and Other Interventional Techniques

SN - 0930-2794

IS - 3

ER -