Who fares worse in an economic downturn, low- or high-paying firms? Different answers to this question imply very different consequences for the costs of recessions. Using U.S. employer-employee data, we find that employment growth at low-paying firms is less cyclically sensitive. High-paying firms grow more quickly in booms and shrink more quickly in busts. We show that while during recessions separations fall in both high-paying and low-paying firms, the decline is stronger among low-paying firms. This is particularly true for separations that are likely voluntary. Our findings thus suggest that downturns hinder upward progression of workers toward higher paying firms – the job ladder partially collapses. Workers at the lowest paying firms are 20% less likely to advance in firm quality (as measured by average pay in a firm) in a bust compared to a boom. Furthermore, workers that join firms in busts compared to booms will on average advance only half as far up the job ladder within the first year, due to both an increased likelihood of matching to a lower paying firm and a reduced probability of moving up once matched. Thus our findings can account for some of the lasting negative impacts on workers forced to search for a job in a downturn, such as displaced workers and recent college graduates.