Spike and slab variable selection: Frequentist and bayesian strategies

Research output: Contribution to journalArticle

335 Citations (Scopus)

Abstract

Variable selection in the linear regression model takes many apparent faces from both frequentist and Bayesian standpoints. In this paper we introduce a variable selection method referred to as a rescaled spike and slab model. We study the importance of prior hierarchical specifications and draw connections to frequentist generalized ridge regression estimation. Specifically, we study the usefulness of continuous bimodal priors to model hypervariance parameters, and the effect scaling has on the posterior mean through its relationship to penalization. Several model selection strategies, some frequentist and some Bayesian in nature, are developed and studied theoretically. We demonstrate the importance of selective shrinkage for effective variable selection in terms of risk misclassification, and show this is achieved using the posterior from a rescaled spike and slab model. We also show how to verify a procedure's ability to reduce model uncertainty in finite samples using a specialized forward selection strategy. Using this tool, we illustrate the effectiveness of rescaled spike and slab models in reducing model uncertainty.

Original languageEnglish
Pages (from-to)730-773
Number of pages44
JournalAnnals of Statistics
Volume33
Issue number2
DOIs
StatePublished - Apr 1 2005
Externally publishedYes

Fingerprint

Variable Selection
Spike
Model Uncertainty
Hierarchical Prior
Posterior Mean
Regression Estimation
Ridge Regression
Misclassification
Penalization
Bimodal
Shrinkage
Linear Regression Model
Model Selection
Model
Face
Scaling
Specification
Verify
Strategy
Variable selection

Keywords

  • Generalized ridge regression
  • Hypervariance
  • Model averaging
  • Model uncertainty
  • Ordinary least squares
  • Penalization
  • Rescaling
  • Shrinkage
  • Stochastic variable selection
  • Zcut

ASJC Scopus subject areas

  • Mathematics(all)
  • Statistics and Probability

Cite this

Spike and slab variable selection : Frequentist and bayesian strategies. / Ishwaran, Hemant; Rao, Jonnagadda S.

In: Annals of Statistics, Vol. 33, No. 2, 01.04.2005, p. 730-773.

Research output: Contribution to journalArticle

@article{d2d8b1819b8a4d8ba1b7f5592456c159,
title = "Spike and slab variable selection: Frequentist and bayesian strategies",
abstract = "Variable selection in the linear regression model takes many apparent faces from both frequentist and Bayesian standpoints. In this paper we introduce a variable selection method referred to as a rescaled spike and slab model. We study the importance of prior hierarchical specifications and draw connections to frequentist generalized ridge regression estimation. Specifically, we study the usefulness of continuous bimodal priors to model hypervariance parameters, and the effect scaling has on the posterior mean through its relationship to penalization. Several model selection strategies, some frequentist and some Bayesian in nature, are developed and studied theoretically. We demonstrate the importance of selective shrinkage for effective variable selection in terms of risk misclassification, and show this is achieved using the posterior from a rescaled spike and slab model. We also show how to verify a procedure's ability to reduce model uncertainty in finite samples using a specialized forward selection strategy. Using this tool, we illustrate the effectiveness of rescaled spike and slab models in reducing model uncertainty.",
keywords = "Generalized ridge regression, Hypervariance, Model averaging, Model uncertainty, Ordinary least squares, Penalization, Rescaling, Shrinkage, Stochastic variable selection, Zcut",
author = "Hemant Ishwaran and Rao, {Jonnagadda S}",
year = "2005",
month = "4",
day = "1",
doi = "10.1214/009053604000001147",
language = "English",
volume = "33",
pages = "730--773",
journal = "Annals of Statistics",
issn = "0090-5364",
publisher = "Institute of Mathematical Statistics",
number = "2",

}

TY - JOUR

T1 - Spike and slab variable selection

T2 - Frequentist and bayesian strategies

AU - Ishwaran, Hemant

AU - Rao, Jonnagadda S

PY - 2005/4/1

Y1 - 2005/4/1

N2 - Variable selection in the linear regression model takes many apparent faces from both frequentist and Bayesian standpoints. In this paper we introduce a variable selection method referred to as a rescaled spike and slab model. We study the importance of prior hierarchical specifications and draw connections to frequentist generalized ridge regression estimation. Specifically, we study the usefulness of continuous bimodal priors to model hypervariance parameters, and the effect scaling has on the posterior mean through its relationship to penalization. Several model selection strategies, some frequentist and some Bayesian in nature, are developed and studied theoretically. We demonstrate the importance of selective shrinkage for effective variable selection in terms of risk misclassification, and show this is achieved using the posterior from a rescaled spike and slab model. We also show how to verify a procedure's ability to reduce model uncertainty in finite samples using a specialized forward selection strategy. Using this tool, we illustrate the effectiveness of rescaled spike and slab models in reducing model uncertainty.

AB - Variable selection in the linear regression model takes many apparent faces from both frequentist and Bayesian standpoints. In this paper we introduce a variable selection method referred to as a rescaled spike and slab model. We study the importance of prior hierarchical specifications and draw connections to frequentist generalized ridge regression estimation. Specifically, we study the usefulness of continuous bimodal priors to model hypervariance parameters, and the effect scaling has on the posterior mean through its relationship to penalization. Several model selection strategies, some frequentist and some Bayesian in nature, are developed and studied theoretically. We demonstrate the importance of selective shrinkage for effective variable selection in terms of risk misclassification, and show this is achieved using the posterior from a rescaled spike and slab model. We also show how to verify a procedure's ability to reduce model uncertainty in finite samples using a specialized forward selection strategy. Using this tool, we illustrate the effectiveness of rescaled spike and slab models in reducing model uncertainty.

KW - Generalized ridge regression

KW - Hypervariance

KW - Model averaging

KW - Model uncertainty

KW - Ordinary least squares

KW - Penalization

KW - Rescaling

KW - Shrinkage

KW - Stochastic variable selection

KW - Zcut

UR - http://www.scopus.com/inward/record.url?scp=22944460748&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=22944460748&partnerID=8YFLogxK

U2 - 10.1214/009053604000001147

DO - 10.1214/009053604000001147

M3 - Article

AN - SCOPUS:22944460748

VL - 33

SP - 730

EP - 773

JO - Annals of Statistics

JF - Annals of Statistics

SN - 0090-5364

IS - 2

ER -