TY  - RPRT
        AU  - Ale, Sonia
AU  - Islam, Md Shafiqul
AU  - Lusher, Lester
AU  - Osman, Huda
AU  - Stenstrom, Jacob
        TI  - When Samples Shape Significance
        PY  - 2026/Aug/
        PB  - Institute of Labor Economics (IZA)
        CY  - Bonn
        T2  - IZA Discussion Paper
        IS  - 18892
        UR  - https://www.iza.org/publications/dp18892
        AB  - Random sampling yields unbiased estimates, yet sampling variation alone can produce incorrect conclusions. Using a setting where repeated draws from the same data-generating process are observable, we replicate findings from 24 papers in top economics journals and re-estimate each result using 40 independent resamples. Across 3,640 resampled results, t-statistics shrink by 31%, frequently leading to lost significance. Proxies for researcher degrees of freedom and sampling noise predict reduced significance. Our exercise provides novel evidence quantifying the effect of sampling variability on empirical research and highlights how statistical evidence can vary even when holding research design fixed.
        KW  - sampling variation
KW  - statistical significance
KW  - reproducibility
KW  - publication bias
KW  - Google Trends
        ER  -