Justin T. Pickett
Sociological Science January 27, 2025
10.15195/v12.a5
Abstract
Forster and Neugebauer’s (2024) invalidation study is invalid. Their conclusion that factorial survey (FS) experiments “are not suited for studying hiring behavior” (P. 901) is unjustified, because their claim that they conducted a field experiment (FE) and FS with “nearly identical” designs is false (P. 891). The two experiments included: (1) different factor levels (for three factors), (2) different unvalidated applicant names (to manipulate ethnicity), (3) different applicant photos, (4) different fixed factors (e.g., applicant stories about moving), and (5) different experimental settings (e.g., testing, instrumentation, and conditions of anonymity). In the current article, I discuss each of these major design differences and explain why it invalidates Forster and Neugebauer’s (2024) comparison of their FE and FS findings. I conclude by emphasizing that social scientists are better served by asking why FE and FS findings sometimes differ than by assuming that any difference in findings across the experimental designs invalidates FS.
Forster and Neugebauer’s (2024) invalidation study is invalid. Their conclusion that factorial survey (FS) experiments “are not suited for studying hiring behavior” (P. 901) is unjustified, because their claim that they conducted a field experiment (FE) and FS with “nearly identical” designs is false (P. 891). The two experiments included: (1) different factor levels (for three factors), (2) different unvalidated applicant names (to manipulate ethnicity), (3) different applicant photos, (4) different fixed factors (e.g., applicant stories about moving), and (5) different experimental settings (e.g., testing, instrumentation, and conditions of anonymity). In the current article, I discuss each of these major design differences and explain why it invalidates Forster and Neugebauer’s (2024) comparison of their FE and FS findings. I conclude by emphasizing that social scientists are better served by asking why FE and FS findings sometimes differ than by assuming that any difference in findings across the experimental designs invalidates FS.
 | This work is licensed under a Creative Commons Attribution 4.0 International License. |
- Citation: Pickett, T. Justin. 2025. “Invalidating Factorial Survey Experiments Using Invalid Comparisons Is Bad Practice: Learning from Forster and Neugebauer (2024)” Sociological Science 12:97-105.
- Received: September 27, 2024
- Accepted: October 1, 2024
- Editors: Arnout van de Rijt, Stephen Vaisey
- DOI: 10.15195/v12.a5