Navigation

You are here: Home › Field Experiment

Tag Archives | Field Experiment

Factorial Survey Experiments to Predict Real-World Behavior: A Cautionary Tale from Hiring Studies

By on September 24, 2024 in Articles

Andrea G. Forster, Martin Neugebauer

Sociological Science September 24, 2024
10.15195/v11.a32

Abstract

PDF (198 views)
0 Citation


Factorial surveys (FSs) are increasingly used to predict real-world decisions. However, there is a paucity of research assessing whether these predictions are valid and, if so, under what conditions. In this preregistered study, we sent out N = 3,002 applications to job vacancies in Germany and measured real-world responses. Eight weeks later, we presented nearly identical applicant profiles to the same employers as a part of an FS. To explore the conditions under which FSs provide valid behavioral predictions, we varied the topic sensitivity and tested whether behavioral predictions were more successful after filtering out respondents who gave socially desirable answers or did not exert sufficient effort when answering FS vignettes. Across conditions, the FS results did not correspond well with the real-world benchmark. We conclude that researchers must exercise caution when using FSs to study (hiring) behavior.
Creative Commons LicenseThis work is licensed under a Creative Commons Attribution 4.0 International License.

Andrea G. Forster: Utrecht University, Heidelberglaan 8, 3584 CS Utrecht, The Netherlands
E-mail: a.g.forster@uu.nl

Martin Neugebauer: Karlsruhe University of Education, Bismarckstr. 10, 76133 Karlsruhe, Germany
E-mail: martin.neugebauer@ph-karlsruhe.de

Acknowledgements: Both authors contributed equally to this study. We would like to thank Lukas Zielinski, Stefan Gunzelmann, Tim Skroblien, Pablo Neitzsch, and Franz Geiger for their help with the design of the experiments and the collection of the data. Furthermore, we would like to thank Katrin Auspurg, Annabell Daniel, Tamara Gutfleisch, Knut Petzold, and Katharina Stückradt as well as 16 professional experts (recruiters and job councelors) for their feedback on our experimental design and materials. Finally, we would like to thank the participants of ECSR 2022, ACES 2022, DGS 2022, the ISOL paper seminar, the Research Colloquium Sociology (University of Bern), and the Research Colloquium Analytical Sociology (LMU Munich) for their feedback on earlier versions of this article. This research was funded by the German Federal Ministry of Education grant number 16PX21011.

Supplemental Materials

Reproducibility Package: The code and data needed to reproduce the analyses are available at the Open Science Framework: https://osf.io/x2tcp/.

  • Citation: Forster, G. Andrea and Martin Neugebauer. 2024. “Factorial Survey Experiments to Predict Real-World Behavior: A Cautionary Tale from Hiring Studies.” Sociological Science 11: 886-906.
  • Received: April 26, 2024
  • Accepted: August 23, 2024
  • Editors: Arnout van de Rijt, Stephen Vaisey
  • DOI: 10.15195/v11.a32


0

Feasible Peer Effects: Experimental Evidence for Deskmate Effects on Educational Achievement and Inequality

By on November 6, 2023 in Articles

Tamás Keller, Felix Elwert

Sociological Science November 6, 2023
10.15195/v10.a28

Abstract

PDF (542 views)
0 Citation


Schools routinely employ seating charts to influence educational outcomes. Dependable evidence for the causal effects of seating charts on students’ achievement levels and inequality, however, is scarce. We executed a large pre-registered field experiment to estimate causal peer effects on students’ test scores and grades by randomizing the seating charts of 195 classrooms (N=3,365 students). We found that neither sitting next to a deskmate with higher prior achievement nor sitting next to a female deskmate affected learning outcomes on average. However, we also found that sitting next to the highest-achieving deskmates improved the educational outcomes of the lowest-achieving students; and sitting next to the lowest-achieving deskmates lowered the educational outcomes of the highest-achieving students. Therefore, compared to random seating charts, achievement-discordant seating charts would decrease inequality; whereas achievement concordant seating charts would increase inequality. We discuss policy implications.
Creative Commons LicenseThis work is licensed under a Creative Commons Attribution 4.0 International License.

Tamás Keller: HUN-REN Centre for Social Sciences, Computational Social Science – Research Center for Educational and Network Studies, Budapest, Hungary HUN-REN Centre for Economic and Regional Studies, Institute of Economics, Budapest, Hungary. TÁRKI Social Research Institute, Budapest, Hungary
E-mail: keller.tamas@tk.hu

Felix Elwert: Department of Sociology & Department of Biostatistics and Medical Informatics, University of Wisconsin-Madison
E-mail: elwert@wisc.edu

Acknowledgements: The authors thank Carlo Barone, Dorottya Baross, Steven Durlauf, Edina Gábor, Eric Grodsky, Judit Kerek, Gábor Kertesi, Gábor Kézdi, Andreas Kotsadam, Xinran Li, Károly Takács, Jeffrey Smith, and audiences at the University of Wisconsin-Madison, the University of Chicago, ISA RC28 Spring Meeting in Frankfurt am Main, the Annual Meeting of the International Network of Analytical Sociologists (INAS) in St. Petersburg, the Meeting of the Economics of Education Association (AEDE) in Las Palmas, the Universita Cattolica del Sacro Cuore, Milan, and the European Research Network on Transitions in Youth (TiY) in Mannheim for valuable discussions. This research was funded by grants from the Hungarian National Research, Development, and Innovation Office (NKFIH), grant number K-135766; a János Bolyai Research Scholarship of the Hungarian Academy of Sciences, #BO/00569/21/9; the New National Excellence Program of the Ministry for Culture and Innovation, #ÚNKP- 23-5-CORVINUS-149; and a Romnes Fellowship and a Vilas Midcareer Faculty Award, University of Wisconsin-Madison. Direct correspondence to Tamás Keller (keller.tamas@tk.hu) and Felix Elwert (elwert@wisc.edu).

Supplemental Material

  • Citation: Keller, Tamás, and Felix Elwert. 2023. “Feasible Peer Effects: Experimental Evidence for Deskmate Effects on Educational Achievement and Inequality” Sociological Science 7: 806-829.
  • Received: July 12, 2023
  • Accepted: May 2, 2023
  • Editors: Arnout van de Rijt, Peter Bearman
  • DOI: 10.15195/v10.a28


0

Does Unprecedented Mass Immigration Fuel Ethnic Discrimination? A Two-Wave Field Experiment in the German Housing Market

By on October 10, 2023 in Articles

Katrin Auspurg, Renate Lorenz, Andreas Schneck

Sociological Science October 10, 2023
10.15195/v10.a23

Abstract

PDF (675 views)
0 Citation


Literature suggests that sudden mass immigration can fuel xenophobic attitudes. However, there is a lack of reliable evidence on hostile actions, such as discrimination. In this study, we leverage the unexpected mass immigration of refugees to Germany in 2015 in combination with a two-wave field experiment to study the effect of immigration on ethnic discrimination. In 2015/2016, political and social tensions in the Middle East and North Africa led to a historic mass migration to European countries. We carried out a large-scale field experiment on ethnic housing market discrimination in Germany (paired e-mail correspondence test with ~5,000 e-mail applications to rental housing units in each wave) shortly before this “European refugee crisis” (1st wave). We repeated this experiment at the peak of the crisis (2nd wave of our experiment). By taking advantage of the unexpected refugee immigration between the two waves of our experiment and the quasi-random allocation of refugees across regions for causal identification, we find no credible evidence that the large influx of refugees changed the extent of ethnic discrimination of Turks in the rental housing market. This result holds regardless of the extent to which regions within Germany were already accustomed to immigration before the refugee crisis.
Creative Commons LicenseThis work is licensed under a Creative Commons Attribution 4.0 International License.

Katrin Auspurg: LMU Munich, Department of Sociology
E-mail: katrin.auspurg@lmu.de

Renate Lorenz: LMU Munich, Department of Sociology
E-mail: renate.lorenz@soziologie.uni-muenchen.de

Andreas Schneck: LMU Munich, Department of Sociology
E-mail: andreas.schneck@lmu.de

Acknowledgements: We thank the participants of the Annual Conference of Experimental Sociology (ACES) 2021, the Research Colloquium of the Doctoral School of Social Sciences at the University of Trento (2022) and the Social Science Research Colloquium at the TU Kaiserslautern (2022) for helpful suggestions. We are grateful for comments on earlier versions we received from Madison Garrett. Maximilian Sonnauer helped us compiling the database for the field experiments.

Supplemental Material

  • Citation: Auspurg, Katrin, Renate Lorenz, and Andreas Schneck. 2023. “Does Unprecedented Mass Immigration Fuel Ethnic Discrimination? A Two-Wave Field Experiment in the German Housing Market.” Sociological Science 10: 640-666.
  • Received: December 17, 2022
  • Accepted: April 17, 2023
  • Editors: Arnout van de Rijt, Maria Abascal
  • DOI: 10.15195/v10.a23


0

Success-Breeds-Success in Collective Political Behavior: Evidence from a Field Experiment

By on October 31, 2016 in Articles

Arnout van de Rijt, Idil Afife Akin, Robb Willer, Matthew Feinberg

Sociological Science, October 31, 2016
DOI 10.15195/v3.a41

Abstract
PDF (3520 views)
0 Citation

Scholars have proposed that the emergence of political movements is highly pathdependent, such that early mobilization successes may lead to disproportionately greater eventual success. This article replicates a unique field experiment testing for positive feedback in internet petition signing (van de Rijt et al. 2014). The prior study found no significant effect of signatures bestowed by the experimenters on the signing rate of 200 online petitions posted to a political petitions website (http://www.change.org), but this may have lacked power because of its sample size and variation across petitions. We report on results of a new field experiment in which we posted 400 petitions differing only in tightly controlled ways to the same website, varying the number of experimentally bestowed signatures across a wider range than in the original experiment. Subsequent petition signing increased monotonically with the treatment, confirming the presence of positive feedback. These results support the existence of success-breeds-success dynamics in the mobilization of collective political behavior, confirming that early success can increase the attractiveness of collective action to potential supporters. However, while significant, the effect of prior signatures was small, suggesting that cumulative advantage effects resulting from popularity metrics may play a minor role in collective action outcomes.

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution 4.0 International License.

Arnout van de Rijt: Department of Sociology, Utrecht University; Department of Sociology, Stony Brook University; Institute for Advanced Computational Science, Stony Brook University
Email: arnoutvanderijt@gmail.com

Idil Afife Akin: Department of Sociology, Stony Brook University
Email: idilakin@gmail.com

Robb Willer: Department of Sociology, Stanford University
Email: willer@stanford.edu

Matthew Feinberg: Rotman School of Management, University of Toronto
Email: matthewfeinberg1@gmail.com

Acknowledgements: We thank the editors and consulting editors for helpful feedback on our initial manuscript, Crystal Redekopp and Michael Claffey for their organizational and technical support of the experiment, and the http://www.change.org team for allowing and hosting our experiment. This work was supported by National Science Foundation grant SES-1340122 (to A.v.d.R.).

Supplemental Material

  • Citation: van de Rijt, Arnout, Idil Afife Akin, Robb Willer, and Matthew Feinberg. 2016. “Success-Breeds-Success in Collective Political Behavior: Evidence from a Field Experiment.” Sociological Science 3: 940-950.
  • Received: August 17, 2016
  • Accepted: September 26, 2016
  • Editors: Delia Baldassarri
  • DOI: 10.15195/v3.a41


0
SiteLock