In March 2020, shelter-in-place and social-distancing policies have been enforced or recommended all over the world to fight the COVID-19 pandemic. However, strict containment is hardly achievable in low-income countries, as large parts of population are forced to continue income-generating activities to escape extreme poverty or hunger. To assess the trade-off between poverty and a higher risk of catching COVID-19, we use regional mobility to work and poverty rates across 241 regions of 9 countries from Latin America and Africa. With a difference-in-difference approach around the time of lockdown announcements, we measure the differential time variation in work mobility between high and low-poverty regions. We find that the degree of work mobility reduction is significantly driven by the intensity of poverty. Consistently, human movements vary significantly more between poverty levels when it come to work rather than less vital activities. We also estimate how higher poverty rates translate into a faster spread of COVID-19 cases through the channel of work mobility.