We show that ideologically motivated disrespect triggers social processes helping individuals protect their self-image and social image, causing ideological and affective polarization as by-products. If even small shares of the population are less respected by one side of the ideological spectrum than the other, ideological views are polarized in equilibrium for almost everyone, including those equally respected by all. Although everyone socializes only with the like-minded in equilibrium, homophily is not assumed but emerges endogenously. The sets of characteristics disrespected by each side determine segregation and polarization patterns, while the degree of polarization depends strongly on exposure to non-peers’ views.