# Data-driven Calibration of Penalties for Least-Squares Regression

Sylvain Arlot, Pascal Massart.

Year: 2009, Volume: 10, Issue: 10, Pages: 245−279

#### Abstract

Penalization procedures often suffer from their dependence on multiplying factors, whose optimal values are either unknown or hard to estimate from data. We propose a completely data-driven calibration algorithm for these parameters in the least-squares regression framework, without assuming a particular shape for the penalty. Our algorithm relies on the concept of minimal penalty, recently introduced by BirgĂ© and Massart (2007) in the context of penalized least squares for Gaussian homoscedastic regression. On the positive side, the minimal penalty can be evaluated from the data themselves, leading to a data-driven estimation of an optimal penalty which can be used in practice; on the negative side, their approach heavily relies on the homoscedastic Gaussian nature of their stochastic framework.

The purpose of this paper is twofold: stating a more general
heuristics for designing a data-driven penalty (the *slope
heuristics*) and proving that it works for penalized least-squares
regression with a random design, even for heteroscedastic non-Gaussian
data. For technical reasons, some exact mathematical results will be
proved only for regressogram bin-width selection. This is at least a
first step towards further results, since the approach and the method
that we use are indeed general.