Performance and estimation of the true error rate of classification rules built with additional information. An application to a cancer trial

Abstract

Classification rules that incorporate additional information usually present in discrimination problems are receiving certain attention during the last years as they perform better than the usual rules in poor discrimination problems. Fern´andez et al (2006) proved that these rules have a lower unconditional misclassification probability than the usual Fisher’s rule but they did not consider the estimation of the conditional error probability when a training sample is given (the so-called true error rate) which is a very interesting parameter in practice.

In this paper we consider the problem of estimating the true error rate of these classification rules in the classical topic of discrimination among two normal populations. We prove theoretical results on the apparent error rate of the rules that expose the need of new estimators of their true error rate. Our proposal is to also consider the additional information in the definition of the true error rate estimators. We propose four such new estimators. Two of them are defined incorporating the additional information into the leave-one-out-bootstrap. The other two are the corresponding cross-validation after bootstrap versions. We compare these new estimators with the usual ones in a simulation study and in a cancer trial application, showing the very good behavior, in terms of mean square error, of the leave-one-out bootstrap estimators that incorporate the available additional information.