Tag Archives | Hiring

Factorial Survey Experiments to Predict Real-World Behavior: A Cautionary Tale from Hiring Studies

Andrea G. Forster, Martin Neugebauer

Sociological Science September 24, 2024
10.15195/v11.a32


Factorial surveys (FSs) are increasingly used to predict real-world decisions. However, there is a paucity of research assessing whether these predictions are valid and, if so, under what conditions. In this preregistered study, we sent out N = 3,002 applications to job vacancies in Germany and measured real-world responses. Eight weeks later, we presented nearly identical applicant profiles to the same employers as a part of an FS. To explore the conditions under which FSs provide valid behavioral predictions, we varied the topic sensitivity and tested whether behavioral predictions were more successful after filtering out respondents who gave socially desirable answers or did not exert sufficient effort when answering FS vignettes. Across conditions, the FS results did not correspond well with the real-world benchmark. We conclude that researchers must exercise caution when using FSs to study (hiring) behavior.
Creative Commons LicenseThis work is licensed under a Creative Commons Attribution 4.0 International License.

Andrea G. Forster: Utrecht University, Heidelberglaan 8, 3584 CS Utrecht, The Netherlands
E-mail: a.g.forster@uu.nl

Martin Neugebauer: Karlsruhe University of Education, Bismarckstr. 10, 76133 Karlsruhe, Germany
E-mail: martin.neugebauer@ph-karlsruhe.de

Acknowledgements: Both authors contributed equally to this study. We would like to thank Lukas Zielinski, Stefan Gunzelmann, Tim Skroblien, Pablo Neitzsch, and Franz Geiger for their help with the design of the experiments and the collection of the data. Furthermore, we would like to thank Katrin Auspurg, Annabell Daniel, Tamara Gutfleisch, Knut Petzold, and Katharina Stückradt as well as 16 professional experts (recruiters and job councelors) for their feedback on our experimental design and materials. Finally, we would like to thank the participants of ECSR 2022, ACES 2022, DGS 2022, the ISOL paper seminar, the Research Colloquium Sociology (University of Bern), and the Research Colloquium Analytical Sociology (LMU Munich) for their feedback on earlier versions of this article. This research was funded by the German Federal Ministry of Education grant number 16PX21011.

Supplemental Materials

Reproducibility Package: The code and data needed to reproduce the analyses are available at the Open Science Framework: https://osf.io/x2tcp/.

  • Citation: Forster, G. Andrea and Martin Neugebauer. 2024. “Factorial Survey Experiments to Predict Real-World Behavior: A Cautionary Tale from Hiring Studies.” Sociological Science 11: 886-906.
  • Received: April 26, 2024
  • Accepted: August 23, 2024
  • Editors: Arnout van de Rijt, Stephen Vaisey
  • DOI: 10.15195/v11.a32


0

Do Some Countries Discriminate More than Others? Evidence from 97 Field Experiments of Racial Discrimination in Hiring

Lincoln Quillian, Anthony Heath, Devah Pager, Arnfinn H. Midtbøen, Fenella Fleischmann, Ole Hexel

Sociological Science, June 17, 2019
10.15195/v6.a18


Comparing levels of discrimination across countries can provide a window into large-scale social and political factors often described as the root of discrimination. Because of difficulties in measurement, however, little is established about variation in hiring discrimination across countries. We address this gap through a formal meta-analysis of 97 field experiments of discrimination incorporating more than 200,000 job applications in nine countries in Europe and North America. We find significant discrimination against nonwhite natives in all countries in our analysis; discrimination against white immigrants is present but low. However, discrimination rates vary strongly by country: In high-discrimination countries, white natives receive nearly twice the callbacks of nonwhites; in low-discrimination countries, white natives receive about 25 percent more. France has the highest discrimination rates, followed by Sweden. We find smaller differences among Great Britain, Canada, Belgium, the Netherlands, Norway, the United States, and Germany. These findings challenge several conventional macro-level theories of discrimination.
Creative Commons LicenseThis work is licensed under a Creative Commons Attribution 4.0 International License.

Lincoln Quillian: Department of Sociology, Northwestern University
E-mail: l-quillian@northwestern.edu

Anthony Heath: Centre for Social Investigation, Nuffield College
E-mail: anthony.heath@nuffield.ox.ac.uk

Devah Pager: Deceased, formerly Department of Sociology, Harvard University

Arnfinn H. Midtbøen: Institute for Social Research, Oslo, Norway
E-mail: a.h.midtboen@samfunnsforskning.no

Fenella Fleischmann: Interdisciplinary Social Science, Utrecht University
E-mail: F.Fleischmann@uu.nl

Ole Hexel: Department of Sociology, Northwestern University, and Observatoire Sociologique du Changement, Sciences Po, Paris, France
E-mail: ole.hexel@u.northwestern.edu

Acknowledgements: We have received financial support for this project from the Russell Sage Foundation and the Institute for Policy Research at Northwestern University. We thank Larry Hedges for methodological advice. We dedicate this article to Devah Pager, who learned a little from us and taught us much more.

  • Citation: Quillian, Lincoln, Anthony Heath, Devah Pager, Arnfinn H. Midtbøen, Fenella Fleischmann, and Ole Hexel. 2019. “Do Some Countries Discriminate More than Others? Evidence from 97 Field Experiments of Racial Discrimination in Hiring.” Sociological Science 6: 467-496.
  • Received: March 7, 2019
  • Accepted: April 23, 2019
  • Editors: Jesper Sørensen, Olav Sorenson
  • DOI: 10.15195/v6.a18


16

The Labor Market Value of Taste: An Experimental Study of Class Bias in U.S. Employment

Kyla Thomas

Sociological Science, September 12, 2018
10.15195/v5.a24


This article investigates cultural forms of class bias in the middle-income U.S. labor market. Results from an audit study of employment discrimination in four U.S. cities reveal that cultural signals of class, when included in résumés, have a systematic effect on the callback rates of women applying to customer-facing jobs. For these women, displays of highbrow taste—the cultural signals of a higher-class background—generate significantly higher rates of employer callback than displays of lowbrow taste—the cultural signals of a lower-class background. Meanwhile, cultural signals of class have no systematic effect on the callback rates of male and/or non–customer-facing job applicants. Results from a survey-experimental study of 1,428 U.S. hiring managers suggest that these differing patterns of employer callback may be explained by the positive effect of higher-class cultural signals on perceptions of polish and competence and their negative effect on perceptions of warmth.
Creative Commons LicenseThis work is licensed under a Creative Commons Attribution 4.0 International License.

Kyla Thomas: Center for Economic and Social Research, University of Southern California
E-mail: kylathom@usc.edu

Acknowledgements: I would like to thank Paul DiMaggio for his intellectual guidance and support as well as Viviana Zelizer, Devah Pager, Susan Fiske, David Pedulla, Patrick Ishizuka, René Flores, and participants of the University of Michigan’s Inequality and Family Working Group for their valuable insights and feedback. This research was supported by the Fahs-Beck Fund for Research and Experimentation, Princeton University’s Center for the Study of Social Organization, and an Andrew W. Mellon Foundation/American Council of Learned Societies Dissertation Completion Fellowship.

  • Citation: Thomas, Kyla. 2018. “The Labor Market Value of Taste: An Experimental Study of Class Bias in U.S. Employment.” Sociological Science 5: 562-595.
  • Received: November 17, 2017
  • Accepted: July 15, 2018
  • Editors: Jesper Sørensen, Kim Weeden
  • DOI: 10.15195/v5.a24


0

At the Expense of Quality

Brittany M. Bond, Tatiana Labuzova, Roberto M. Fernandez

Sociological Science, June 28, 2018
10.15195/v5.a17


Many organizations use employee referral programs to incentivize employees to refer potential applicants from their social networks. Employers frequently offer a monetary bonus to employees who refer an applicant, and this is often contingent on whether the person is then hired and retained for a given length of time. In deciding whether to refer someone, referrers face a potential role conflict, as they need to balance their motivations for helping connections find job opportunities with concerns regarding their reputations with their employers. To the extent that monetary incentives shift an employee’s considerations away from finding the best matches for the employer, referral bonuses may increase the chances that lower-quality candidates are referred. Using a survey vignette experiment, we find that even a small referral bonus increases the likelihood that referrers will refer lower-quality candidates, and they are more likely to refer people they do not know well. We further discuss theoretical and practical implications regarding the efficiency of incentivized referral programs in producing quality applicant pools for employers.
Creative Commons LicenseThis work is licensed under a Creative Commons Attribution 4.0 International License.

Brittany M. Bond: Sloan School of Management, Massachusetts Institute of Technology
E-mail: bbond@mit.edu

Tatiana Labuzova: Sloan School of Management, Massachusetts Institute of Technology
E-mail: labuzova@mit.edu

Roberto M. Fernandez: Sloan School of Management, Massachusetts Institute of Technology
E-mail: robertof@mit.edu

Acknowledgements: We thank our colleagues at the Massachusetts Institute of Technology Sloan School of Management and elsewhere for their feedback on earlier versions of this article. We have also benefitted from help, advice, and feedback from Matthew Amengual, Rhett Andrew Brymer, Santiago Campero, John Carroll, Emilio Castilla, Minjae Kim, Ezra Zuckerman Sivan, Heather Yang, and participants in the Economic Sociology Working Group at MIT Sloan and the Ninth Annual Meeting of the People and Organizations Conference at The Wharton School.

  • Citation: Bond, Brittany M., Tatiana Labuzova, and Roberto M. Fernandez. 2018. “At the Expense of Quality.” Sociological Science 5: 380-401.
  • Received: March 27, 2018
  • Accepted: April 17, 2018
  • Editors: Jesper Sørensen, Mario Small
  • DOI: 10.15195/v5.a17

0
SiteLock