This paper relies on a large-scale sample of genotyped individuals linked with detailed register data in Denmark to investigate the context-dependence of genetic influences on human capital formation. We show that the returns to genetic endowments, measured by a polygenic score for educational attainment, are significantly attenuated by childhood disadvantage. We replicate the findings in a within-family analysis, where we exploit exogenous genetic variation across siblings to control for unobserved family influences. We also explore gender differences in the context-dependence of genetic influences and find the attenuation effect of childhood disadvantage on educational attainment to be significantly stronger for males than for females. We show our findings extend to a representative sample of the Danish population. Our results highlight an important mechanism driving the persistence of disadvantage across generations. We show that children who experience childhood disadvantage are not able to fully realize their educational potential, even in the context of the generous Danish welfare-state.