There has been a recent rapid increase in immigration into Europe, specifically in the form of refugees and asylum seekers. This raises a range of social challenges and a particular focus is education and school systems. A growing body of research investigates the impact of immigrants on native test score performance. In practice this reports very mixed results and a difficulty is that immigrant groups are often pooled together due to data restrictions. We return to this issue using Norwegian register data that allows us to distinguish refugees from other immigrants. Using narrow within-school, within-family comparisons combined with the Norwegian refugee settlement system we demonstrate marked negative effects of refugee children on the test score performance of their native school children classmates. These effects are simply not present for other immigrants, and stem primarily from refugee children who themselves are most at risk of low performance. These negative effects are concentrated on students at most risk of underperformance, boys and children from lower educated backgrounds, and may reflect a lack of compensatory inputs at schools.