Statistical power in the journal of research in music education (2000-2010): A retrospective power analysis

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

During the last half of the 20th century, researchers became increasingly aware of a tendency in much social science research to lack sufficient statistical power. Low power can cause studies to be unable to reliably detect true effects in a population (Type II error) and to overestimate the practical significance of small effects when they are detected. Despite cautionary writings and the widespread availability of helpful computational tools, systemically underpowered research persists in the majority of contemporary academic journals. While the Journal of Research in Music Education is arguably the profession's most prominent platform for empirical research, the degree to which this issue may be present has not yet been examined. Results found that the research articles published in the journal between 2000-2010 share similarities with other prominent journals in education and educational psychology in that, on average, the statistical power of the designs fell well below commonly accepted thresholds. Survey research in music education gener-ally fared the best, being more adequately powered (.72) than many other academic disciplines, although still below the de facto standard of .80. By contrast, the average power of designs involv-ing the manipulation of experimental conditions was much lower (.55). These figures represent best-case scenarios. Reports often did not contain all relevant information necessary to determine some common hidden drains to power, requiring some concessions to be made during calculation that likely overestimate the final results. Suggestions for maximizing statistical power are offered as well as a discussion of alternatives to classical null hypotheses significance testing.

Original languageEnglish (US)
Pages (from-to)43-59
Number of pages17
JournalBulletin of the Council for Research in Music Education
Issue number201
DOIs
StatePublished - Jun 1 2014

Fingerprint

music lessons
educational psychology
survey research
concession
allies
manipulation
empirical research
social science
profession
Music Education
scenario
cause
lack
education

ASJC Scopus subject areas

  • Music
  • Education

Cite this

@article{416dd3d8b4634314832651e85d986917,
title = "Statistical power in the journal of research in music education (2000-2010): A retrospective power analysis",
abstract = "During the last half of the 20th century, researchers became increasingly aware of a tendency in much social science research to lack sufficient statistical power. Low power can cause studies to be unable to reliably detect true effects in a population (Type II error) and to overestimate the practical significance of small effects when they are detected. Despite cautionary writings and the widespread availability of helpful computational tools, systemically underpowered research persists in the majority of contemporary academic journals. While the Journal of Research in Music Education is arguably the profession's most prominent platform for empirical research, the degree to which this issue may be present has not yet been examined. Results found that the research articles published in the journal between 2000-2010 share similarities with other prominent journals in education and educational psychology in that, on average, the statistical power of the designs fell well below commonly accepted thresholds. Survey research in music education gener-ally fared the best, being more adequately powered (.72) than many other academic disciplines, although still below the de facto standard of .80. By contrast, the average power of designs involv-ing the manipulation of experimental conditions was much lower (.55). These figures represent best-case scenarios. Reports often did not contain all relevant information necessary to determine some common hidden drains to power, requiring some concessions to be made during calculation that likely overestimate the final results. Suggestions for maximizing statistical power are offered as well as a discussion of alternatives to classical null hypotheses significance testing.",
author = "Corin Overland",
year = "2014",
month = "6",
day = "1",
doi = "10.5406/bulcouresmusedu.201.0043",
language = "English (US)",
pages = "43--59",
journal = "Bulletin of the Council for Research in Music Education",
issn = "0010-9894",
publisher = "University of Illinois at Urbana-Champaign",
number = "201",

}

TY - JOUR

T1 - Statistical power in the journal of research in music education (2000-2010)

T2 - A retrospective power analysis

AU - Overland, Corin

PY - 2014/6/1

Y1 - 2014/6/1

N2 - During the last half of the 20th century, researchers became increasingly aware of a tendency in much social science research to lack sufficient statistical power. Low power can cause studies to be unable to reliably detect true effects in a population (Type II error) and to overestimate the practical significance of small effects when they are detected. Despite cautionary writings and the widespread availability of helpful computational tools, systemically underpowered research persists in the majority of contemporary academic journals. While the Journal of Research in Music Education is arguably the profession's most prominent platform for empirical research, the degree to which this issue may be present has not yet been examined. Results found that the research articles published in the journal between 2000-2010 share similarities with other prominent journals in education and educational psychology in that, on average, the statistical power of the designs fell well below commonly accepted thresholds. Survey research in music education gener-ally fared the best, being more adequately powered (.72) than many other academic disciplines, although still below the de facto standard of .80. By contrast, the average power of designs involv-ing the manipulation of experimental conditions was much lower (.55). These figures represent best-case scenarios. Reports often did not contain all relevant information necessary to determine some common hidden drains to power, requiring some concessions to be made during calculation that likely overestimate the final results. Suggestions for maximizing statistical power are offered as well as a discussion of alternatives to classical null hypotheses significance testing.

AB - During the last half of the 20th century, researchers became increasingly aware of a tendency in much social science research to lack sufficient statistical power. Low power can cause studies to be unable to reliably detect true effects in a population (Type II error) and to overestimate the practical significance of small effects when they are detected. Despite cautionary writings and the widespread availability of helpful computational tools, systemically underpowered research persists in the majority of contemporary academic journals. While the Journal of Research in Music Education is arguably the profession's most prominent platform for empirical research, the degree to which this issue may be present has not yet been examined. Results found that the research articles published in the journal between 2000-2010 share similarities with other prominent journals in education and educational psychology in that, on average, the statistical power of the designs fell well below commonly accepted thresholds. Survey research in music education gener-ally fared the best, being more adequately powered (.72) than many other academic disciplines, although still below the de facto standard of .80. By contrast, the average power of designs involv-ing the manipulation of experimental conditions was much lower (.55). These figures represent best-case scenarios. Reports often did not contain all relevant information necessary to determine some common hidden drains to power, requiring some concessions to be made during calculation that likely overestimate the final results. Suggestions for maximizing statistical power are offered as well as a discussion of alternatives to classical null hypotheses significance testing.

UR - http://www.scopus.com/inward/record.url?scp=84929339233&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84929339233&partnerID=8YFLogxK

U2 - 10.5406/bulcouresmusedu.201.0043

DO - 10.5406/bulcouresmusedu.201.0043

M3 - Article

AN - SCOPUS:84929339233

SP - 43

EP - 59

JO - Bulletin of the Council for Research in Music Education

JF - Bulletin of the Council for Research in Music Education

SN - 0010-9894

IS - 201

ER -