December 2013

IZA DP No. 7874: Mostly Harmless Simulations? On the Internal Validity of Empirical Monte Carlo Studies

substantially revised version forthcoming as "Mostly Harmless Simulations? Using Monte Carlo Studies for Estimator Selection" in: Journal of Applied Econometrics

In this paper we evaluate the premise from the recent literature on Monte Carlo studies that an empirically motivated simulation exercise is informative about the actual ranking of various estimators when applied to a particular problem. We consider two alternative designs and provide an empirical test for both of them. We conclude that a necessary condition for the simulations to be informative about the true ranking is that the treatment effect in simulations must be equal to the (unknown) true effect. This severely limits the usefulness of such procedures, since were the effect known, the procedure would not be necessary.