Output per worker is lower in poor countries than in rich countries, and relatively more so in the agricultural sector. Sorting of heterogeneous workers can contribute to explain this fact if comparative and absolute advantage are aligned in agriculture, implying that average productivity in agriculture increases as the agricultural employment share decreases. We empirically investigate the correlation between comparative and absolute advantage using representative household-level panel data from four Sub-Saharan African countries. Around one third of households engage in both agriculture and non-farming entrepreneurship. We find that more productive farming households are more likely to also engage in non-farm entrepreneurship, allocate more hours to it if they do, and are more likely to enter it if not yet active. All three pieces of evidence imply that comparative and absolute advantage are negatively correlated – misaligned – in agriculture, casting doubt on the importance of selection as a root cause of the agricultural productivity gap.