Estimation of Quantiles for the Sampling Distribution of the rWG within Group Agreement Index

John M. Charnes, Chester Schriesheim

Research output: Contribution to journalArticle

9 Citations (Scopus)

Abstract

The James, Demaree, and Wolf within-group agreement index (rwG) may be used to assess interrater agreement when two or more individuals (“judges”) provide ratings on one or more items (“targets”). Although the James et al. index is increasingly being used to assess whether the aggregation of data is justified for higher levels of analysis, the sampling distribution of rWG has not been delineated, and therefore it has not been possible to conduct tests of the statistical significance of obtained results. As a consequence of this current deficiency in knowledge about rWG, a Monte Carlo simulation program (AGREESIM) was developed to estimate selected quantiles for the sampling distribution of rWG under any user-specified null distribution of judges’ scores. The output is in the form of quantiles and their 95% confidence intervals (showing the precision of the quantile estimates).

Original languageEnglish (US)
Pages (from-to)435-437
Number of pages3
JournalEducational and Psychological Measurement
Volume55
Issue number3
DOIs
StatePublished - 1995

Fingerprint

Sampling Distribution
Quantile
Confidence Intervals
Sampling
Agglomeration
Group
Null Distribution
Statistical Significance
statistical significance
aggregation
Estimate
Confidence interval
Aggregation
Monte Carlo Simulation
confidence
rating
simulation
Target
Output
Monte Carlo simulation

ASJC Scopus subject areas

  • Developmental and Educational Psychology
  • Applied Psychology
  • Applied Mathematics
  • Education

Cite this

Estimation of Quantiles for the Sampling Distribution of the rWG within Group Agreement Index. / Charnes, John M.; Schriesheim, Chester.

In: Educational and Psychological Measurement, Vol. 55, No. 3, 1995, p. 435-437.

Research output: Contribution to journalArticle

@article{2dd51990d9f34788bacdd3be037b083d,
title = "Estimation of Quantiles for the Sampling Distribution of the rWG within Group Agreement Index",
abstract = "The James, Demaree, and Wolf within-group agreement index (rwG) may be used to assess interrater agreement when two or more individuals (“judges”) provide ratings on one or more items (“targets”). Although the James et al. index is increasingly being used to assess whether the aggregation of data is justified for higher levels of analysis, the sampling distribution of rWG has not been delineated, and therefore it has not been possible to conduct tests of the statistical significance of obtained results. As a consequence of this current deficiency in knowledge about rWG, a Monte Carlo simulation program (AGREESIM) was developed to estimate selected quantiles for the sampling distribution of rWG under any user-specified null distribution of judges’ scores. The output is in the form of quantiles and their 95{\%} confidence intervals (showing the precision of the quantile estimates).",
author = "Charnes, {John M.} and Chester Schriesheim",
year = "1995",
doi = "10.1177/0013164495055003007",
language = "English (US)",
volume = "55",
pages = "435--437",
journal = "Educational and Psychological Measurement",
issn = "0013-1644",
publisher = "SAGE Publications Inc.",
number = "3",

}

TY - JOUR

T1 - Estimation of Quantiles for the Sampling Distribution of the rWG within Group Agreement Index

AU - Charnes, John M.

AU - Schriesheim, Chester

PY - 1995

Y1 - 1995

N2 - The James, Demaree, and Wolf within-group agreement index (rwG) may be used to assess interrater agreement when two or more individuals (“judges”) provide ratings on one or more items (“targets”). Although the James et al. index is increasingly being used to assess whether the aggregation of data is justified for higher levels of analysis, the sampling distribution of rWG has not been delineated, and therefore it has not been possible to conduct tests of the statistical significance of obtained results. As a consequence of this current deficiency in knowledge about rWG, a Monte Carlo simulation program (AGREESIM) was developed to estimate selected quantiles for the sampling distribution of rWG under any user-specified null distribution of judges’ scores. The output is in the form of quantiles and their 95% confidence intervals (showing the precision of the quantile estimates).

AB - The James, Demaree, and Wolf within-group agreement index (rwG) may be used to assess interrater agreement when two or more individuals (“judges”) provide ratings on one or more items (“targets”). Although the James et al. index is increasingly being used to assess whether the aggregation of data is justified for higher levels of analysis, the sampling distribution of rWG has not been delineated, and therefore it has not been possible to conduct tests of the statistical significance of obtained results. As a consequence of this current deficiency in knowledge about rWG, a Monte Carlo simulation program (AGREESIM) was developed to estimate selected quantiles for the sampling distribution of rWG under any user-specified null distribution of judges’ scores. The output is in the form of quantiles and their 95% confidence intervals (showing the precision of the quantile estimates).

UR - http://www.scopus.com/inward/record.url?scp=84973799195&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84973799195&partnerID=8YFLogxK

U2 - 10.1177/0013164495055003007

DO - 10.1177/0013164495055003007

M3 - Article

AN - SCOPUS:84973799195

VL - 55

SP - 435

EP - 437

JO - Educational and Psychological Measurement

JF - Educational and Psychological Measurement

SN - 0013-1644

IS - 3

ER -