Our paper documents the importance of ex ante worker heterogeneity for labor market dynamics and for the composition of the unemployment pool over the business cycle. In recessions, the unemployment pool shifts toward workers with higher wages in their previous jobs. Based on administrative data for Germany and two-way worker and firm wage fixed effects, we show that this shift is mainly connected to worker heterogeneity, not to firm heterogeneity. We calibrate a search and matching model with ex ante worker heterogeneity to the estimated relative residual wage dispersion across worker fixed-effect groups. We show that a lower idiosyncratic match-specific shock dispersion for high-wage workers is key for the larger relative fluctuations of their separation rate as well as for the positive co-movement between prior wages and fixed effects of unemployed workers with aggregate unemployment. We argue that firm-based explanations, such as cyclical financial frictions, are unlikely to be key drivers for the documented empirical patterns.