A sizeable literature analyses how immigration affects attitudes towards migrants and discusses differences between socio-economic groups and their potential correlation with perceived concerns about labour market competition. Against the background of the large-scale influx of refugees into Germany between 2015 and 2016, this paper uses data from a unique and representative survey of the German population to assess whether respondents express fears of job loss due to immigration. We focus on the importance of perceptions of migrants' ability to do one's job in relation to these fears. Moreover, we compare concerns about refugees with those about EU migrants and propose several hypotheses. Our findings indicate that: (i) Respondents are more likely to view EU migrants as potential competitors in the labour market. (ii) Workers in blue-collar occupations and without tertiary education are more likely to view migrants as potential competitors on the labour market. (iii) The perception of potential competition from migrants strongly predicts fear of job loss. Once we control for this perception, occupation and skill levels are no longer significantly related to the probability of reporting fear of job loss. Moreover, there are no longer significant differences between the two migrant groups. (iv) Anti-migrant sentiments are also associated with concerns about job loss.