Reliability, precision, and measurement in the context of data from ability tests, surveys, and assessments

W. P. Fisher, Batya Elbaum, A. Coulter

Research output: Contribution to journalArticle

7 Citations (Scopus)

Abstract

Reliability coefficients indicate the proportion of total variance attributable to differences among measures separated along a quantitative continuum by a testing, survey, or assessment instrument. Reliability is usually considered to be influenced by both the internal consistency of a data set and the number of items, though textbooks and research papers rarely evaluate the extent to which these factors independently affect the data in question. Probabilistic formulations of the requirements for unidimensional measurement separate consistency from error by modelling individual response processes instead of group-level variation. The utility of this separation is illustrated via analyses of small sets of simulated data, and of subsets of data from a 78-item survey of over 2,500 parents of children with disabilities. Measurement reliability ultimately concerns the structural invariance specified in models requiring sufficient statistics, parameter separation, unidimensionality, and other qualities that historically have made quantification simple, practical, and convenient for end users. The paper concludes with suggestions for a research program aimed at focusing measurement research more on the calibration and wide dissemination of tools applicable to individuals, and less on the statistical study of inter-variable relations in large data sets.

Original languageEnglish
Article number012036
JournalJournal of Physics: Conference Series
Volume238
DOIs
StatePublished - Dec 1 2010

Fingerprint

disabilities
textbooks
set theory
suggestion
invariance
proportion
statistics
continuums
formulations
requirements
coefficients

ASJC Scopus subject areas

  • Physics and Astronomy(all)

Cite this

Reliability, precision, and measurement in the context of data from ability tests, surveys, and assessments. / Fisher, W. P.; Elbaum, Batya; Coulter, A.

In: Journal of Physics: Conference Series, Vol. 238, 012036, 01.12.2010.

Research output: Contribution to journalArticle

@article{25a22c54793848139f0c88514cddecbc,
title = "Reliability, precision, and measurement in the context of data from ability tests, surveys, and assessments",
abstract = "Reliability coefficients indicate the proportion of total variance attributable to differences among measures separated along a quantitative continuum by a testing, survey, or assessment instrument. Reliability is usually considered to be influenced by both the internal consistency of a data set and the number of items, though textbooks and research papers rarely evaluate the extent to which these factors independently affect the data in question. Probabilistic formulations of the requirements for unidimensional measurement separate consistency from error by modelling individual response processes instead of group-level variation. The utility of this separation is illustrated via analyses of small sets of simulated data, and of subsets of data from a 78-item survey of over 2,500 parents of children with disabilities. Measurement reliability ultimately concerns the structural invariance specified in models requiring sufficient statistics, parameter separation, unidimensionality, and other qualities that historically have made quantification simple, practical, and convenient for end users. The paper concludes with suggestions for a research program aimed at focusing measurement research more on the calibration and wide dissemination of tools applicable to individuals, and less on the statistical study of inter-variable relations in large data sets.",
author = "Fisher, {W. P.} and Batya Elbaum and A. Coulter",
year = "2010",
month = "12",
day = "1",
doi = "10.1088/1742-6596/238/1/012036",
language = "English",
volume = "238",
journal = "Journal of Physics: Conference Series",
issn = "1742-6588",
publisher = "IOP Publishing Ltd.",

}

TY - JOUR

T1 - Reliability, precision, and measurement in the context of data from ability tests, surveys, and assessments

AU - Fisher, W. P.

AU - Elbaum, Batya

AU - Coulter, A.

PY - 2010/12/1

Y1 - 2010/12/1

N2 - Reliability coefficients indicate the proportion of total variance attributable to differences among measures separated along a quantitative continuum by a testing, survey, or assessment instrument. Reliability is usually considered to be influenced by both the internal consistency of a data set and the number of items, though textbooks and research papers rarely evaluate the extent to which these factors independently affect the data in question. Probabilistic formulations of the requirements for unidimensional measurement separate consistency from error by modelling individual response processes instead of group-level variation. The utility of this separation is illustrated via analyses of small sets of simulated data, and of subsets of data from a 78-item survey of over 2,500 parents of children with disabilities. Measurement reliability ultimately concerns the structural invariance specified in models requiring sufficient statistics, parameter separation, unidimensionality, and other qualities that historically have made quantification simple, practical, and convenient for end users. The paper concludes with suggestions for a research program aimed at focusing measurement research more on the calibration and wide dissemination of tools applicable to individuals, and less on the statistical study of inter-variable relations in large data sets.

AB - Reliability coefficients indicate the proportion of total variance attributable to differences among measures separated along a quantitative continuum by a testing, survey, or assessment instrument. Reliability is usually considered to be influenced by both the internal consistency of a data set and the number of items, though textbooks and research papers rarely evaluate the extent to which these factors independently affect the data in question. Probabilistic formulations of the requirements for unidimensional measurement separate consistency from error by modelling individual response processes instead of group-level variation. The utility of this separation is illustrated via analyses of small sets of simulated data, and of subsets of data from a 78-item survey of over 2,500 parents of children with disabilities. Measurement reliability ultimately concerns the structural invariance specified in models requiring sufficient statistics, parameter separation, unidimensionality, and other qualities that historically have made quantification simple, practical, and convenient for end users. The paper concludes with suggestions for a research program aimed at focusing measurement research more on the calibration and wide dissemination of tools applicable to individuals, and less on the statistical study of inter-variable relations in large data sets.

UR - http://www.scopus.com/inward/record.url?scp=78651096941&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=78651096941&partnerID=8YFLogxK

U2 - 10.1088/1742-6596/238/1/012036

DO - 10.1088/1742-6596/238/1/012036

M3 - Article

AN - SCOPUS:78651096941

VL - 238

JO - Journal of Physics: Conference Series

JF - Journal of Physics: Conference Series

SN - 1742-6588

M1 - 012036

ER -