This paper analyzes the impact of rainfall fluctuations during the gestational period on health at birth. We concentrate on the semiarid region of Northeastern Brazil to highlight the role of water scarcity as a determinant of early life health. We find that negative rainfall shocks are robustly correlated with higher infant mortality, lower birth weight, and shorter gestation periods. Mortality effects are concentrated on intestinal infections and malnutrition, and are greatly minimized when the local public health infrastructure is sufficiently developed (municipality coverage of piped water and sanitation). We also find that effects are stronger during the fetal period (2nd trimester of gestation), for children born during the dry season, and for mortality in the first 6 months of life. The results seem to be driven by water scarcity per se, and not by reduced agricultural production. Our estimates suggest that expansions in public health infrastructure would be a cost-effective way of reducing the response of infant mortality to rainfall shocks in the Brazilian semiarid.