Food production and distribution is essential for human well-being, but the food sector has experienced a number of difficulties maintaining worker health and productivity during the COVID-19 pandemic. We examine employment status changes of persons recently employed in the U.S. food sector with a focus on food manufacturing and grocery stores. We find that the pandemic significantly reduced the probability of continued active employment for previous workers in both food manufacturing and grocery stores. Individual-level analysis confirms that the COVID-19 infection rate in an individual's local labor market is a strong and significant factor. The employment changes are not just due to unemployment during facility closures. Previous workers increasingly exit the labor force as the severity of the COVID-19 infection rate in their local area worsens. The considerable risk of infection drives many previous food sector workers to stop working altogether. Maintaining worker health and safety is essential for a stable food supply.