Least angle regression

Bradley Efron, Trevor Hastie, Iain Johnstone, Robert Tibshirani, Hemant Ishwaran, Keith Knight, Jean Michel Loubes, Pascal Massart, David Madigan, Greg Ridgeway, Saharon Rosset, J. I. Zhu, Robert A. Stine, Berwin A. Turlach, Sanford Weisberg, Trevor Hastie, Iain Johnstone, Robert Tibshirani

Research output: Contribution to journalArticle

4736 Citations (Scopus)

Abstract

The purpose of model selection algorithms such as All Subsets, Forward Selection and Backward Elimination is to choose a linear model on the basis of the same set of data to which the model will be applied. Typically we have available a large collection of possible covariates from which we hope to select a parsimonious set for the efficient prediction of a response variable. Least Angle Regression (LARS), a new model selection algorithm, is a useful and less greedy version of traditional forward selection methods. Three main properties are derived: (1) A simple modification of the LARS algorithm implements the Lasso, an attractive version of ordinary least squares that constrains the sum of the absolute regression coefficients; the LARS modification calculates all possible Lasso estimates for a given problem, using an order of magnitude less computer time than previous methods. (2) A different LARS modification efficiently implements Forward Stagewise linear regression, another promising new model selection method; this connection explains the similar numerical results previously observed for the Lasso and Stagewise, and helps us understand the properties of both methods, which are seen as constrained versions of the simpler LARS algorithm. (3) A simple approximation for the degrees of freedom of a LARS estimate is available, from which we derive a C p estimate of prediction error; this allows a principled choice among the range of possible LARS estimates. LARS and its variants are computationally efficient: the paper describes a publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates.

Original languageEnglish
Pages (from-to)407-499
Number of pages93
JournalAnnals of Statistics
Volume32
Issue number2
DOIs
StatePublished - Apr 1 2004
Externally publishedYes

Fingerprint

Regression
Angle
Lasso
Model Selection
Regression Estimate
Ordinary Least Squares
Covariates
Prediction Error
Regression Coefficient
Linear regression
Estimate
Elimination
Linear Model
Choose
Degree of freedom
Calculate
Numerical Results
Subset
Prediction
Approximation

Keywords

  • Boosting
  • Coefficient paths
  • Lasso
  • Linear regression
  • Variable selection

ASJC Scopus subject areas

  • Mathematics(all)
  • Statistics and Probability

Cite this

Efron, B., Hastie, T., Johnstone, I., Tibshirani, R., Ishwaran, H., Knight, K., ... Tibshirani, R. (2004). Least angle regression. Annals of Statistics, 32(2), 407-499. https://doi.org/10.1214/009053604000000067

Least angle regression. / Efron, Bradley; Hastie, Trevor; Johnstone, Iain; Tibshirani, Robert; Ishwaran, Hemant; Knight, Keith; Loubes, Jean Michel; Massart, Pascal; Madigan, David; Ridgeway, Greg; Rosset, Saharon; Zhu, J. I.; Stine, Robert A.; Turlach, Berwin A.; Weisberg, Sanford; Hastie, Trevor; Johnstone, Iain; Tibshirani, Robert.

In: Annals of Statistics, Vol. 32, No. 2, 01.04.2004, p. 407-499.

Research output: Contribution to journalArticle

Efron, B, Hastie, T, Johnstone, I, Tibshirani, R, Ishwaran, H, Knight, K, Loubes, JM, Massart, P, Madigan, D, Ridgeway, G, Rosset, S, Zhu, JI, Stine, RA, Turlach, BA, Weisberg, S, Hastie, T, Johnstone, I & Tibshirani, R 2004, 'Least angle regression', Annals of Statistics, vol. 32, no. 2, pp. 407-499. https://doi.org/10.1214/009053604000000067
Efron B, Hastie T, Johnstone I, Tibshirani R, Ishwaran H, Knight K et al. Least angle regression. Annals of Statistics. 2004 Apr 1;32(2):407-499. https://doi.org/10.1214/009053604000000067
Efron, Bradley ; Hastie, Trevor ; Johnstone, Iain ; Tibshirani, Robert ; Ishwaran, Hemant ; Knight, Keith ; Loubes, Jean Michel ; Massart, Pascal ; Madigan, David ; Ridgeway, Greg ; Rosset, Saharon ; Zhu, J. I. ; Stine, Robert A. ; Turlach, Berwin A. ; Weisberg, Sanford ; Hastie, Trevor ; Johnstone, Iain ; Tibshirani, Robert. / Least angle regression. In: Annals of Statistics. 2004 ; Vol. 32, No. 2. pp. 407-499.
@article{8c7c23a4c6bc4efb97ab3e98944bd38d,
title = "Least angle regression",
abstract = "The purpose of model selection algorithms such as All Subsets, Forward Selection and Backward Elimination is to choose a linear model on the basis of the same set of data to which the model will be applied. Typically we have available a large collection of possible covariates from which we hope to select a parsimonious set for the efficient prediction of a response variable. Least Angle Regression (LARS), a new model selection algorithm, is a useful and less greedy version of traditional forward selection methods. Three main properties are derived: (1) A simple modification of the LARS algorithm implements the Lasso, an attractive version of ordinary least squares that constrains the sum of the absolute regression coefficients; the LARS modification calculates all possible Lasso estimates for a given problem, using an order of magnitude less computer time than previous methods. (2) A different LARS modification efficiently implements Forward Stagewise linear regression, another promising new model selection method; this connection explains the similar numerical results previously observed for the Lasso and Stagewise, and helps us understand the properties of both methods, which are seen as constrained versions of the simpler LARS algorithm. (3) A simple approximation for the degrees of freedom of a LARS estimate is available, from which we derive a C p estimate of prediction error; this allows a principled choice among the range of possible LARS estimates. LARS and its variants are computationally efficient: the paper describes a publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates.",
keywords = "Boosting, Coefficient paths, Lasso, Linear regression, Variable selection",
author = "Bradley Efron and Trevor Hastie and Iain Johnstone and Robert Tibshirani and Hemant Ishwaran and Keith Knight and Loubes, {Jean Michel} and Pascal Massart and David Madigan and Greg Ridgeway and Saharon Rosset and Zhu, {J. I.} and Stine, {Robert A.} and Turlach, {Berwin A.} and Sanford Weisberg and Trevor Hastie and Iain Johnstone and Robert Tibshirani",
year = "2004",
month = "4",
day = "1",
doi = "10.1214/009053604000000067",
language = "English",
volume = "32",
pages = "407--499",
journal = "Annals of Statistics",
issn = "0090-5364",
publisher = "Institute of Mathematical Statistics",
number = "2",

}

TY - JOUR

T1 - Least angle regression

AU - Efron, Bradley

AU - Hastie, Trevor

AU - Johnstone, Iain

AU - Tibshirani, Robert

AU - Ishwaran, Hemant

AU - Knight, Keith

AU - Loubes, Jean Michel

AU - Massart, Pascal

AU - Madigan, David

AU - Ridgeway, Greg

AU - Rosset, Saharon

AU - Zhu, J. I.

AU - Stine, Robert A.

AU - Turlach, Berwin A.

AU - Weisberg, Sanford

AU - Hastie, Trevor

AU - Johnstone, Iain

AU - Tibshirani, Robert

PY - 2004/4/1

Y1 - 2004/4/1

N2 - The purpose of model selection algorithms such as All Subsets, Forward Selection and Backward Elimination is to choose a linear model on the basis of the same set of data to which the model will be applied. Typically we have available a large collection of possible covariates from which we hope to select a parsimonious set for the efficient prediction of a response variable. Least Angle Regression (LARS), a new model selection algorithm, is a useful and less greedy version of traditional forward selection methods. Three main properties are derived: (1) A simple modification of the LARS algorithm implements the Lasso, an attractive version of ordinary least squares that constrains the sum of the absolute regression coefficients; the LARS modification calculates all possible Lasso estimates for a given problem, using an order of magnitude less computer time than previous methods. (2) A different LARS modification efficiently implements Forward Stagewise linear regression, another promising new model selection method; this connection explains the similar numerical results previously observed for the Lasso and Stagewise, and helps us understand the properties of both methods, which are seen as constrained versions of the simpler LARS algorithm. (3) A simple approximation for the degrees of freedom of a LARS estimate is available, from which we derive a C p estimate of prediction error; this allows a principled choice among the range of possible LARS estimates. LARS and its variants are computationally efficient: the paper describes a publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates.

AB - The purpose of model selection algorithms such as All Subsets, Forward Selection and Backward Elimination is to choose a linear model on the basis of the same set of data to which the model will be applied. Typically we have available a large collection of possible covariates from which we hope to select a parsimonious set for the efficient prediction of a response variable. Least Angle Regression (LARS), a new model selection algorithm, is a useful and less greedy version of traditional forward selection methods. Three main properties are derived: (1) A simple modification of the LARS algorithm implements the Lasso, an attractive version of ordinary least squares that constrains the sum of the absolute regression coefficients; the LARS modification calculates all possible Lasso estimates for a given problem, using an order of magnitude less computer time than previous methods. (2) A different LARS modification efficiently implements Forward Stagewise linear regression, another promising new model selection method; this connection explains the similar numerical results previously observed for the Lasso and Stagewise, and helps us understand the properties of both methods, which are seen as constrained versions of the simpler LARS algorithm. (3) A simple approximation for the degrees of freedom of a LARS estimate is available, from which we derive a C p estimate of prediction error; this allows a principled choice among the range of possible LARS estimates. LARS and its variants are computationally efficient: the paper describes a publicly available algorithm that requires only the same order of magnitude of computational effort as ordinary least squares applied to the full set of covariates.

KW - Boosting

KW - Coefficient paths

KW - Lasso

KW - Linear regression

KW - Variable selection

UR - http://www.scopus.com/inward/record.url?scp=3242708140&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=3242708140&partnerID=8YFLogxK

U2 - 10.1214/009053604000000067

DO - 10.1214/009053604000000067

M3 - Article

VL - 32

SP - 407

EP - 499

JO - Annals of Statistics

JF - Annals of Statistics

SN - 0090-5364

IS - 2

ER -