While gender gaps in average math performance are close to zero in developed countries, women are still strongly underrepresented among math high performers. Using data from five successive waves of the Programme for International Student Assessment (PISA), we show that this underrepresentation is more severe in more unequal countries. This relationship holds for a wide range of societal inequalities that are not directly related to gender. It is also observed in other parts of the performance distribution and among various sets of countries, including developing countries. Similar relationships are found in science and reading. Such findings highlight how differences in socio-economic and cultural factors can affect gender gaps in performance.