Comparing the vibration of effects due to model, data pre-processing and sampling uncertainty on a large data set in personality psychology

Klau, Simon; Schönbrodt, Felix D. ; Patel, Chirag; Ioannidis, John; Boulesteix, Anne-Laure und Hoffmann, Sabine (5. Februar 2020): Comparing the vibration of effects due to model, data pre-processing and sampling uncertainty on a large data set in personality psychology. Department of Statistics: Technical Reports, Nr. 232

Abstract

Researchers have great flexibility in the analysis of observational data. If combined with selective reporting and pressure to publish, this flexibility can have devastating consequences on the validity of research findings. We extend the recently proposed vibration of effects approach to provide a framework comparing three main sources of uncertainty which lead to instability in observational associations, namely data pre-processing, model and sampling uncertainty. We analyze their behavior for varying sample sizes for two associations in personality psychology. While all types of vibration show a decrease for increasing sample sizes, data pre-processing and model vibration remain non-negligible, even for a sample of over 80000 participants. The increasing availability of large data sets that are not initially recorded for research purposes can make data pre-processing and model choices very influential. We therefore recommend the framework as a tool for the transparent reporting of the stability of research findings.

• Dokumententyp: Paper
• Keywords: metascience, researcher degrees of freedom, stability, replicability, Big Five
• Fakultät: Mathematik, Informatik und Statistik > Statistik > Technische Reports
• Themengebiete: 100 Philosophie und Psychologie > 150 Psychologie ; 300 Sozialwissenschaften > 310 Statistiken ; 500 Naturwissenschaften und Mathematik > 510 Mathematik