Climate change has increased rainfall uncertainty, leading to greater production risks in agriculture. We examine the gender-differentiated labor impacts of droughts resulting from lower precipitation using unique individual-level panel data for agricultural households in India over half a decade. Accounting for unobserved heterogeneity in individual responses, we find that women's workdays fall by 11% more than men's when a drought occurs, driven by former's lack of diversification to the non-farm sector. Women are less likely to work outside their village and migrate relative to men in response to droughts, and are consequently unable to cope fully with the adverse agricultural productivity shock. Our findings can be explained by social costs emanating from gender norms that constrain women's access to non-farm work opportunities. The results highlight the gendered impact of climate change, potentially exacerbating extant gender gaps in the labor market.