Sparseness vs Estimating Conditional Probabilities: Some Asymptotic Results
Peter L. Bartlett, Ambuj Tewari.
Year: 2007, Volume: 8, Issue: 28, Pages: 775−790
Abstract
One of the nice properties of kernel classifiers such as SVMs is that they often produce sparse solutions. However, the decision functions of these classifiers cannot always be used to estimate the conditional probability of the class label. We investigate the relationship between these two properties and show that these are intimately related: sparseness does not occur when the conditional probabilities can be unambiguously estimated. We consider a family of convex loss functions and derive sharp asymptotic results for the fraction of data that becomes support vectors. This enables us to characterize the exact trade-off between sparseness and the ability to estimate conditional probabilities for these loss functions.