Pitfalls of hypothesis tests and model selection on bootstrap samples: causes and consequences in biometrical applications

Janitza, Silke; Binder, Harald und Boulesteix, Anne-Laure (19. November 2014): Pitfalls of hypothesis tests and model selection on bootstrap samples: causes and consequences in biometrical applications. Department of Statistics: Technical Reports, No.163

Abstract

The bootstrap method has become a widely used tool applied in diverse areas where results based on asymptotic theory are scarce. It can be applied for example for assessing the variance of a statistic, a quantile of interest or for significance testing by resampling from the null hypothesis. Recently some approaches have been proposed in the biometrical field where hypothesis testing or model selection is performed on a bootstrap sample as if it were the original sample. p-values computed from bootstrap samples have been used for example in the statistics and bioinformatics literature for ranking genes with respect to their differential expression, for estimating the variability of p-values and for model stability investigations. Procedures which make use of bootstrapped information criteria are often applied in the model stability investigations and model averaging approaches as well as when estimating the error of model selection procedures which involve tuning parameters. From the literature, however, there is evidence that p-values and model selection criteria evaluated on bootstrap data sets do not adequately represent what would be obtained on the original data or new data drawn from the overall population. We explain the reasons for this and, through the use of a real data set and simulations, we assess the practical impact on procedures relevant to biometrical applications in cases where it has not yet been studied. Moreover, we investigate the behaviour of subsampling (i.e., drawing from a data set without replacement) as a potential alternative solution to the bootstrap for these procedures.

• Item Type: Paper
• Keywords: Bootstrap; Bootstrapped information criteria; Bootstrapped p-values; Bootstrapped test statistic; Tests on bootstrap samples.
• Faculties: Mathematics, Computer Science and Statistics > Statistics > Technical Reports
• Subjects: 500 Science > 500 Science
• Language: English