We combine community-level outcomes of 27 votes about immigration issues in Switzerland with census data to estimate the effect of immigration on natives' attitudes towards immigration. We apply an instrumental variable approach to take potentially endogenous locational choices into account, and we categorize immigrants into two groups according to the cultural values and beliefs of their source country to understand how the cultural distance between natives and immigrants affects this relationship. We find that the share of culturally different immigrants is a significant and sizable determinant of anti-immigration votes, while the presence of culturally similar immigrants does not affect natives' voting behavior at all in most specifications. The cultural distance between immigrant and native residents thus appears crucial in explaining the causal effect of immigration on natives' attitudes towards immigration, and we argue that the differential impact is mainly driven by natives' concerns about compositional amenities. We finally show that the elasticity of the share of right-wing votes in favor of the Swiss People's Party is much more elastic with respect to the share of culturally different immigrants than natives' attitudes themselves, suggesting that the party has disproportionally gained from changes in attitudes caused by immigrant inflows.