Stability enhanced large-margin classifier selection

Wei Sun, Guang Cheng, Yufeng Liu

Research output: Contribution to journalArticle

Abstract

Stability is an important aspect of a classification procedure as unstable predictions can potentially reduce users' trust in a classification system and harm the reproducibility of scientific conclusions. We introduce a concept of classification instability, decision boundary instability (DBI), and incorporate it with the generalization error (GE) as a standard for selecting the most accurate and stable classifier. For this, we implement a two-stage algorithm: (i) select a subset of classifiers whose estimated GEs are not significantly different from the minimal estimated GE among all the candidate classifiers; (ii) take the optimal classifier to be the one achieving the minimal DBI among the subset selected in stage (i). This selection principle applies to both linear and nonlinear classifiers. Large-margin classifiers are used as a prototypical example to illustrate this idea. Our selection method is shown to be consistent in the sense that the optimal classifier simultaneously achieves the minimal GE and the minimal DBI. Various simulations and examples further demonstrate the advantage of our method over alternative approaches.

Original languageEnglish (US)
Pages (from-to)1-25
Number of pages25
JournalStatistica Sinica
Volume28
Issue number1
DOIs
StatePublished - Jan 1 2018

Fingerprint

Margin
Classifier
Generalization Error
Selection Principles
Subset
Reproducibility
Unstable
Prediction
Alternatives
Demonstrate
Simulation

Keywords

  • Asymptotic normality
  • Large-margin
  • Model selection
  • Selection consistency
  • Stability

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Cite this

Stability enhanced large-margin classifier selection. / Sun, Wei; Cheng, Guang; Liu, Yufeng.

In: Statistica Sinica, Vol. 28, No. 1, 01.01.2018, p. 1-25.

Research output: Contribution to journalArticle

Sun, Wei ; Cheng, Guang ; Liu, Yufeng. / Stability enhanced large-margin classifier selection. In: Statistica Sinica. 2018 ; Vol. 28, No. 1. pp. 1-25.
@article{2a4069ec017b49fcb6bb566d237be2f6,
title = "Stability enhanced large-margin classifier selection",
abstract = "Stability is an important aspect of a classification procedure as unstable predictions can potentially reduce users' trust in a classification system and harm the reproducibility of scientific conclusions. We introduce a concept of classification instability, decision boundary instability (DBI), and incorporate it with the generalization error (GE) as a standard for selecting the most accurate and stable classifier. For this, we implement a two-stage algorithm: (i) select a subset of classifiers whose estimated GEs are not significantly different from the minimal estimated GE among all the candidate classifiers; (ii) take the optimal classifier to be the one achieving the minimal DBI among the subset selected in stage (i). This selection principle applies to both linear and nonlinear classifiers. Large-margin classifiers are used as a prototypical example to illustrate this idea. Our selection method is shown to be consistent in the sense that the optimal classifier simultaneously achieves the minimal GE and the minimal DBI. Various simulations and examples further demonstrate the advantage of our method over alternative approaches.",
keywords = "Asymptotic normality, Large-margin, Model selection, Selection consistency, Stability",
author = "Wei Sun and Guang Cheng and Yufeng Liu",
year = "2018",
month = "1",
day = "1",
doi = "10.5705/ss.202016.0260",
language = "English (US)",
volume = "28",
pages = "1--25",
journal = "Statistica Sinica",
issn = "1017-0405",
publisher = "Institute of Statistical Science",
number = "1",

}

TY - JOUR

T1 - Stability enhanced large-margin classifier selection

AU - Sun, Wei

AU - Cheng, Guang

AU - Liu, Yufeng

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Stability is an important aspect of a classification procedure as unstable predictions can potentially reduce users' trust in a classification system and harm the reproducibility of scientific conclusions. We introduce a concept of classification instability, decision boundary instability (DBI), and incorporate it with the generalization error (GE) as a standard for selecting the most accurate and stable classifier. For this, we implement a two-stage algorithm: (i) select a subset of classifiers whose estimated GEs are not significantly different from the minimal estimated GE among all the candidate classifiers; (ii) take the optimal classifier to be the one achieving the minimal DBI among the subset selected in stage (i). This selection principle applies to both linear and nonlinear classifiers. Large-margin classifiers are used as a prototypical example to illustrate this idea. Our selection method is shown to be consistent in the sense that the optimal classifier simultaneously achieves the minimal GE and the minimal DBI. Various simulations and examples further demonstrate the advantage of our method over alternative approaches.

AB - Stability is an important aspect of a classification procedure as unstable predictions can potentially reduce users' trust in a classification system and harm the reproducibility of scientific conclusions. We introduce a concept of classification instability, decision boundary instability (DBI), and incorporate it with the generalization error (GE) as a standard for selecting the most accurate and stable classifier. For this, we implement a two-stage algorithm: (i) select a subset of classifiers whose estimated GEs are not significantly different from the minimal estimated GE among all the candidate classifiers; (ii) take the optimal classifier to be the one achieving the minimal DBI among the subset selected in stage (i). This selection principle applies to both linear and nonlinear classifiers. Large-margin classifiers are used as a prototypical example to illustrate this idea. Our selection method is shown to be consistent in the sense that the optimal classifier simultaneously achieves the minimal GE and the minimal DBI. Various simulations and examples further demonstrate the advantage of our method over alternative approaches.

KW - Asymptotic normality

KW - Large-margin

KW - Model selection

KW - Selection consistency

KW - Stability

UR - http://www.scopus.com/inward/record.url?scp=85040098853&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85040098853&partnerID=8YFLogxK

U2 - 10.5705/ss.202016.0260

DO - 10.5705/ss.202016.0260

M3 - Article

AN - SCOPUS:85040098853

VL - 28

SP - 1

EP - 25

JO - Statistica Sinica

JF - Statistica Sinica

SN - 1017-0405

IS - 1

ER -