Apesteguia and Palacios-Huerta (forthcoming) report for a sample of 129 shootouts from various seasons in ten different competitions that teams kicking first in soccer penalty shootouts win significantly more often than teams kicking second. Collecting data for the entire history of six major soccer competitions we cannot replicate their result. Teams kicking first win only 53.4% of 262 shootouts in our data, which is not significantly different from random. Our findings have two implications: (1) Apesteguia and Palacios-Huerta's results are not generally robust. (2) Using specific subsamples without a coherent criterion for data selection might lead to non-representative findings.