Use of the estimated intraclass correlation for correcting differences in effect size by level

Soyeon Ahn, Nicholas Myers, Ying Jin

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

In a meta-analysis of intervention or group comparison studies, researchers often encounter the circumstance in which the standardized mean differences (d-effect sizes) are computed at multiple levels (e. g., individual vs. cluster). Cluster-level d-effect sizes may be inflated and, thus, may need to be corrected using the intraclass correlation (ICC) before being combined with individual-level d-effect sizes. The ICC value, however, is seldom reported in primary studies and, thus, may need to be computed from other sources. This article proposes a method for estimating the ICC value from the reported standard deviations within a particular meta-analysis (i. e., estimated ICC) when an appropriate default ICC value (Hedges, 2009b) is unavailable. A series of simulations provided evidence that the proposed method yields an accurate and precise estimated ICC value, which can then be used for correct estimation of a d-effect size. The effects of other pertinent factors (e. g., number of studies) were also examined, followed by discussion of related limitations and future research in this area.

Original languageEnglish
Pages (from-to)490-502
Number of pages13
JournalBehavior Research Methods
Volume44
Issue number2
DOIs
StatePublished - Jun 1 2012

Fingerprint

Meta-Analysis
Research Personnel
Effect Size
Meta-analysis

Keywords

  • Intraclass correlation (ICC)
  • Level of analysis
  • Meta-analysis
  • Multilevel
  • Standardized mean difference

ASJC Scopus subject areas

  • Psychology(all)
  • Psychology (miscellaneous)
  • Experimental and Cognitive Psychology
  • Arts and Humanities (miscellaneous)
  • Developmental and Educational Psychology

Cite this

Use of the estimated intraclass correlation for correcting differences in effect size by level. / Ahn, Soyeon; Myers, Nicholas; Jin, Ying.

In: Behavior Research Methods, Vol. 44, No. 2, 01.06.2012, p. 490-502.

Research output: Contribution to journalArticle

@article{8b8cc8f076a74b0997cd81b980e0cc76,
title = "Use of the estimated intraclass correlation for correcting differences in effect size by level",
abstract = "In a meta-analysis of intervention or group comparison studies, researchers often encounter the circumstance in which the standardized mean differences (d-effect sizes) are computed at multiple levels (e. g., individual vs. cluster). Cluster-level d-effect sizes may be inflated and, thus, may need to be corrected using the intraclass correlation (ICC) before being combined with individual-level d-effect sizes. The ICC value, however, is seldom reported in primary studies and, thus, may need to be computed from other sources. This article proposes a method for estimating the ICC value from the reported standard deviations within a particular meta-analysis (i. e., estimated ICC) when an appropriate default ICC value (Hedges, 2009b) is unavailable. A series of simulations provided evidence that the proposed method yields an accurate and precise estimated ICC value, which can then be used for correct estimation of a d-effect size. The effects of other pertinent factors (e. g., number of studies) were also examined, followed by discussion of related limitations and future research in this area.",
keywords = "Intraclass correlation (ICC), Level of analysis, Meta-analysis, Multilevel, Standardized mean difference",
author = "Soyeon Ahn and Nicholas Myers and Ying Jin",
year = "2012",
month = "6",
day = "1",
doi = "10.3758/s13428-011-0153-1",
language = "English",
volume = "44",
pages = "490--502",
journal = "Behavior Research Methods",
issn = "1554-351X",
publisher = "Springer New York",
number = "2",

}

TY - JOUR

T1 - Use of the estimated intraclass correlation for correcting differences in effect size by level

AU - Ahn, Soyeon

AU - Myers, Nicholas

AU - Jin, Ying

PY - 2012/6/1

Y1 - 2012/6/1

N2 - In a meta-analysis of intervention or group comparison studies, researchers often encounter the circumstance in which the standardized mean differences (d-effect sizes) are computed at multiple levels (e. g., individual vs. cluster). Cluster-level d-effect sizes may be inflated and, thus, may need to be corrected using the intraclass correlation (ICC) before being combined with individual-level d-effect sizes. The ICC value, however, is seldom reported in primary studies and, thus, may need to be computed from other sources. This article proposes a method for estimating the ICC value from the reported standard deviations within a particular meta-analysis (i. e., estimated ICC) when an appropriate default ICC value (Hedges, 2009b) is unavailable. A series of simulations provided evidence that the proposed method yields an accurate and precise estimated ICC value, which can then be used for correct estimation of a d-effect size. The effects of other pertinent factors (e. g., number of studies) were also examined, followed by discussion of related limitations and future research in this area.

AB - In a meta-analysis of intervention or group comparison studies, researchers often encounter the circumstance in which the standardized mean differences (d-effect sizes) are computed at multiple levels (e. g., individual vs. cluster). Cluster-level d-effect sizes may be inflated and, thus, may need to be corrected using the intraclass correlation (ICC) before being combined with individual-level d-effect sizes. The ICC value, however, is seldom reported in primary studies and, thus, may need to be computed from other sources. This article proposes a method for estimating the ICC value from the reported standard deviations within a particular meta-analysis (i. e., estimated ICC) when an appropriate default ICC value (Hedges, 2009b) is unavailable. A series of simulations provided evidence that the proposed method yields an accurate and precise estimated ICC value, which can then be used for correct estimation of a d-effect size. The effects of other pertinent factors (e. g., number of studies) were also examined, followed by discussion of related limitations and future research in this area.

KW - Intraclass correlation (ICC)

KW - Level of analysis

KW - Meta-analysis

KW - Multilevel

KW - Standardized mean difference

UR - http://www.scopus.com/inward/record.url?scp=84861352446&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84861352446&partnerID=8YFLogxK

U2 - 10.3758/s13428-011-0153-1

DO - 10.3758/s13428-011-0153-1

M3 - Article

C2 - 21960061

AN - SCOPUS:84861352446

VL - 44

SP - 490

EP - 502

JO - Behavior Research Methods

JF - Behavior Research Methods

SN - 1554-351X

IS - 2

ER -