We use cookies to provide you with the best possible website experience. This includes cookies that are necessary for the operation of the site, as well as cookies used for anonymous statistics, comfort settings, or displaying personalized content. You can decide which categories you want to allow. Please note that depending on your settings, some features of the website may not be available.

Cookie settings

These necessary cookies are required to enable the core functionality of the website. Opting out of these cookies is not possible.

cb-enable
This cookie stores the user's cookie consent status for the current domain. Expiry: 1 year.
laravel_session
Stores the session ID to recognize the user when the page reloads and to restore their login session. Expiry: 2 hours.
XSRF-TOKEN
Provides CSRF protection for forms. Expiry: 2 hours.
IZA Discussion Paper No. 17744
February 2025
The Sources of Researcher Variation in Economics
Nick Huntington-Klein, Claus C. Portner, Sebastian Gallegos, et al.

We use a rigorous three-stage many-analysts design to assess how different researcher decisions—specifically data cleaning, research design, and the interpretation of a policy question—affect the variation in estimated treatment effects. A total of 146 research teams each completed the same causal inference task three times each: first with few constraints, then using a shared research design, and finally with pre-cleaned data in addition to a specified design. We find that even when analyzing the same data, teams reach different conclusions. In the first stage, the interquartile range (IQR) of the reported policy effect was 3.1 percentage points, with substantial outliers. Surprisingly, the second stage, which restricted research design choices, exhibited slightly higher IQR (4.0 percentage points), largely attributable to imperfect adherence to the prescribed protocol. By contrast, the final stage, featuring standardized data cleaning, narrowed variation in estimated effects, achieving an IQR of 2.4 percentage points. Reported sample sizes also displayed significant convergence under more restrictive conditions, with the IQR dropping from 295,187 in the first stage to 29,144 in the second, and effectively zero by the third. Our findings underscore the critical importance of data cleaning in shaping applied microeconomic results and highlight avenues for future replication efforts.

Communications
Mark Fallak
mark.fallak@liser.lu
+352 585-855-526
World of Labour
Olga Nottmeyer
olga.nottmeyer@liser.lu
+352 585-855-501
Network Coordination
Christina Gathmann
christina.gathmann@liser.lu

The IZA@LISER Network is a global community of scholars dedicated to excellence in labor economics and related fields, now coordinated at the Luxembourg Institute of Socio-Economic Research (LISER) following its transition from Bonn.

About IZA@LISER Network
Contact
IZA Network (Current Site Operator):

Luxembourg Institute of Socio-Economic Research (LISER)
11, Porte des Sciences
Maison des Sciences Humaines
L-4366 Esch-sur-Alzette / Belval, Luxembourg

IZA Institute (In Liquidation):

Forschungsinstitut zur Zukunft der Arbeit GmbH i. L.
Schaumburg-Lippe-Str. 5-9, 53113 Bonn. Germany
Phone: +49 228 3894-0 | Fax: +49 228 3894-510
E-Mail: info@iza.org | Web: www.iza.org
Represented by: Martin T. Clemens (Liquidator)