Avoiding 'data snooping' in multilevel and mixed effects models

David Afshartous, Michael Wolf

Research output: Contribution to journalArticlepeer-review

12 Scopus citations


Multilevel or mixed effects models are commonly applied to hierarchical data. The level 2 residuals, which are otherwise known as random effects, are often of both substantive and diagnostic interest. Substantively, they are frequently used for institutional comparisons or rankings. Diagnostically, they are used to assess the model assumptions at the group level. Inference on the level 2 residuals, however, typically does not account for 'data snooping', i.e. for the harmful effects of carrying out a multitude of hypothesis tests at the same time. We provide a very general framework that encompasses both of the following inference problems: inference on the 'absolute' level 2 residuals to determine which are significantly different from 0, and inference on any prespecified number of pairwise comparisons. Thus, the user has the choice of testing the comparisons of interest. As our methods are flexible with respect to the estimation method that is invoked, the user may choose the desired estimation method accordingly. We demonstrate the methods with the London education authority data, the wafer data and the National Educational Longitudinal Study data.

Original languageEnglish (US)
Pages (from-to)1035-1059
Number of pages25
JournalJournal of the Royal Statistical Society. Series A: Statistics in Society
Issue number4
StatePublished - Oct 2007


  • Data snooping
  • Hierarchical linear models
  • Hypothesis testing
  • Pairwise comparisons
  • Random effects
  • Rankings

ASJC Scopus subject areas

  • Statistics and Probability
  • Social Sciences (miscellaneous)
  • Economics and Econometrics
  • Statistics, Probability and Uncertainty


Dive into the research topics of 'Avoiding 'data snooping' in multilevel and mixed effects models'. Together they form a unique fingerprint.

Cite this