Religious and ethnic minorities across the world face partisan treatment with regard to provision of public goods, either as outcome of discriminatory practices or due to historical antecedents, such as the caste and religious divides in India. In several districts of West Bengal in India concentration of religious minorities, namely Muslims is higher than state and country-level averages. We measures access to public goods in rural West Bengal for different strata of minority concentration. Using Least Square, Generalized Linear Models and Blinder-Oaxaca decomposition, we find evidence of strong horizontal inequality against Muslims in terms of access to public goods. Blinder-Oaxaca decomposition shows that Muslims in larger concentrations face poorer access to infrastructure, health and transport facilities.