Given the critical role of neurocognitive development in early life, this study assesses how racial differences in early-life circumstances are collectively and individually associated with racial disparities in late-life cognition. Leveraging uniquely rich information on life history from the U.S. Health and Retirement Study for non-Hispanic White (White) and non-Hispanic Black (Black) Americans 50 years or older, we employ the Blinder-Oaxaca method to decompose racial gaps in cognitive outcomes into early-life educational experiences, cohort, regional, financial, health, trauma, family relationship, demographic and genetic factors. Overall, differences in early-life circumstances are associated with 61.5% and 82.3% of the racial disparities in cognitive score and impairment, respectively. Early-life educational experience is associated with 35.2% of the disparities in cognitive score and 48.6% in cognitive impairment. Notably, school racial segregation (all segregated schooling before college) is associated with 28.8%-39.7% of the racial disparities in cognition. Policies that improve educational equity have the potential to reduce racial disparities in cognition into older ages. Clinicians may leverage early-life circumstances to promote the screening, prevention, and interventions of cognitive impairment more efficiently, thereby promoting health equity.