The 2015 refugee crisis in Europe fueled anti-immigration sentiment in receiving areas, with potential unintended consequences for refugee integration. We investigate the heterogeneity of political backlash across Italian municipalities in the aftermath of the crisis and assess the role played by local conditions at the time of refugees' settlement, distinguishing between baseline economic and cultural factors. By leveraging the quasi-random dispersal policy and using causal forests, we find that the impact of refugee exposure on anti-immigration backlash is significantly higher in more affluent areas, with more bonding social capital. The opposite holds in contexts where there is meaningful intergroup contact with former immigrants (e.g mixed marriages). We exploit this pattern of heterogeneity to evaluate a matching model to optimally assign refugees to locations and deliver policy implications for novel refugee resettlement schemes that minimize anti-immigration backlash.