We examine the long-term relationship between childhood circumstances and cognitive aging. In particular, we differentiate the level of cognitive deficit from the rate of cognitive decline. Applying a linear mixed-effect model to three waves of China Health and Retirement Longitudinal Surveys (CHARLS 2011, 2013, 2015) and matching cognitive outcomes to CHARLS Life History Survey (2014), we find that key domains of childhood circumstances, including family socioeconomic status (SES), neighborhood cohesion, friendship and health conditions, are significantly associated with both the level of cognitive deficit and the rate of decline. In contrast, childhood neighborhood safety only affects the level of cognitive deficit. Childhood relationship with mother only affects the rate of cognitive decline. The effects of adverse childhood circumstances are generally larger on level of cognitive deficit than on rate of cognitive decline. Moreover, education plays a more important role in mediating the relationships compared to other later-life factors. These findings suggest that exposure to disadvantaged childhood circumstances can exacerbate cognitive deficit as well as cognitive decline over time, which may be partially ameliorated by educational attainment.