A comparison of AUC estimators in small-sample studies
Antti Airola, Tapio Pahikkala, Willem Waegeman, Bernard De Baets, Tapio Salakoski;
JMLR W&CP 8:3-13, 2010.
Reliable estimation of the classification performance of learned predictive
models is difficult, when working in the small sample setting. When dealing with
biological data it is often the case that separate test data cannot be afforded.
Cross-validation is in this case a typical strategy for
estimating the performance. Recent results, further supported by experimental
evidence presented in this article, show that many standard approaches to
cross-validation suffer from extensive bias or variance when the area under
ROC curve (AUC) is used as performance measure. We advocate
the use of leave-pair-out cross-validation (LPOCV) for performance estimation,
as it avoids many of these problems. A method previously proposed by us
can be used to efficiently calculate this estimate for regularized
least-squares (RLS) based learners.