Longer term exposure to high poverty neighbourhoods can affect individual socio-economic outcomes later in life. Previous research has shown strong path dependence in individual neighbourhood histories. A growing literature shows that the neighbourhood histories of people is linked to the neighbourhoods of their childhood and parental characteristics. To better understand intergenerational transmission of living in deprived neighbourhoods it is important to distinguish between inherited disadvantage (socio-economic position) and contextual disadvantage (environmental context in which children grow up). The objective of this paper is to come to a better understanding of the effects of inherited and contextual disadvantage on the neighbourhood careers of children once they have left the parental home. We use a quasi-experimental family design exploiting sibling relationships, including real sibling pairs, and "synthetic siblings" who are used as a control group. Using rich register data from Sweden we find that real siblings live more similar lives in terms of neighbourhood experiences during their independent residential career than synthetic sibling pairs. This difference reduces over time. Real siblings are still less different than synthetic pairs but the difference gets smaller with time, indicating a quicker attenuation of the family effect on residential outcomes than the neighbourhood effect.