We investigate the impact of family co-residence structure and the allocation of major childcare responsibility across generations on a child's cognitive development. Using data from China, we find that children living in multigenerational families generally perform better in their cognitive tests after controlling for other factors. This result holds only for elementary school children, but not for middle school children. However, children who live only with their parents and children who live only with their grandparents (the left-behind children) do not show a significant difference in their cognitive performance. Moreover, we find that the effect of family environment differs between boys and girls. Girls from multigenerational families with grandparents as the main caregiver generally do better than other girls; while for boys, three-generation co-residence has a positive impact regardless of who the main caregiver is. Additionally, there is some evidence that the co-residence and childcare arrangements respond to the cognitive performance of girls more than boys. Our exploration of behavioral factors as potential operating mechanisms in explaining our findings indicates that the influences of family environment are complex and subtle.