TY  - RPRT
        AU  - Huntington-Klein, Nick
AU  - Arenas, Andreu
AU  - Beam, Emily A.
AU  - Bertoni, Marco
AU  - Bloem, Jeffrey R.
AU  - Burli, Pralhad
AU  - Chen, Naibin
AU  - Greico, Paul
AU  - Ekpe, Godwin
AU  - Pugatch, Todd
AU  - Saavedra, Martin
AU  - Stopnitzky, Yaniv
        TI  - The Influence of Hidden Researcher Decisions in Applied Microeconomics
        PY  - 2020/May/
        PB  - Institute of Labor Economics (IZA)
        CY  - Bonn
        T2  - IZA Discussion Paper
        IS  - 13233
        UR  - https://www.iza.org/index.php/publications/dp13233
        AB  - Researchers make hundreds of decisions about data collection, preparation, and analysis in their research. We use a many-analysts approach to measure the extent and impact of these decisions. Two published causal empirical results are replicated by seven replicators each. We find large differences in data preparation and analysis decisions, many of which would not likely be reported in a publication. No two replicators reported the same sample size. Statistical significance varied across replications, and for one of the studies the effect's sign varied as well. The standard deviation of estimates across replications was 3-4 times the typical reported standard error.
        KW  - metascience
KW  - replication
KW  - research
        ER  -