Tag Archives | Validation Study

Factorial Survey Experiments to Predict Real-World Behavior: A Cautionary Tale from Hiring Studies

Andrea G. Forster, Martin Neugebauer

Sociological Science September 24, 2024
10.15195/v11.a32


Factorial surveys (FSs) are increasingly used to predict real-world decisions. However, there is a paucity of research assessing whether these predictions are valid and, if so, under what conditions. In this preregistered study, we sent out N = 3,002 applications to job vacancies in Germany and measured real-world responses. Eight weeks later, we presented nearly identical applicant profiles to the same employers as a part of an FS. To explore the conditions under which FSs provide valid behavioral predictions, we varied the topic sensitivity and tested whether behavioral predictions were more successful after filtering out respondents who gave socially desirable answers or did not exert sufficient effort when answering FS vignettes. Across conditions, the FS results did not correspond well with the real-world benchmark. We conclude that researchers must exercise caution when using FSs to study (hiring) behavior.
Creative Commons LicenseThis work is licensed under a Creative Commons Attribution 4.0 International License.

Andrea G. Forster: Utrecht University, Heidelberglaan 8, 3584 CS Utrecht, The Netherlands
E-mail: a.g.forster@uu.nl

Martin Neugebauer: Karlsruhe University of Education, Bismarckstr. 10, 76133 Karlsruhe, Germany
E-mail: martin.neugebauer@ph-karlsruhe.de

Acknowledgements: Both authors contributed equally to this study. We would like to thank Lukas Zielinski, Stefan Gunzelmann, Tim Skroblien, Pablo Neitzsch, and Franz Geiger for their help with the design of the experiments and the collection of the data. Furthermore, we would like to thank Katrin Auspurg, Annabell Daniel, Tamara Gutfleisch, Knut Petzold, and Katharina Stückradt as well as 16 professional experts (recruiters and job councelors) for their feedback on our experimental design and materials. Finally, we would like to thank the participants of ECSR 2022, ACES 2022, DGS 2022, the ISOL paper seminar, the Research Colloquium Sociology (University of Bern), and the Research Colloquium Analytical Sociology (LMU Munich) for their feedback on earlier versions of this article. This research was funded by the German Federal Ministry of Education grant number 16PX21011.

Supplemental Materials

Reproducibility Package: The code and data needed to reproduce the analyses are available at the Open Science Framework: https://osf.io/x2tcp/.

  • Citation: Forster, G. Andrea and Martin Neugebauer. 2024. “Factorial Survey Experiments to Predict Real-World Behavior: A Cautionary Tale from Hiring Studies.” Sociological Science 11: 886-906.
  • Received: April 26, 2024
  • Accepted: August 23, 2024
  • Editors: Arnout van de Rijt, Stephen Vaisey
  • DOI: 10.15195/v11.a32


0
SiteLock