Long-term exposure to extreme temperatures could threaten individuals' mental health and psychological wellbeing. This study aims to investigate the long-term impact of cumulative exposure to extreme temperature. Differently from existing literature, we define extreme temperature exposure in relative terms based on local temperature patterns. Combining the China Health and Retirement Longitudinal Study and environmental data from the U.S. National Oceanic and Atmospheric Administration from 2011 to 2015, this study demonstrates that heat and cold exposure days in the past year significantly increase the measured depression level of adults over age 45 by 1.75 and 3.00 per cent, respectively, controlling for the city, year, and individual fixed effects. The effect is heterogeneous across three components of depression symptoms as well as age, gender, and areas of residency, and air conditioning and heating equipment are effective in alleviating the adverse impact of heat and cold exposure. The estimation is robust and consistent across a variety of temperature measurements and model modifications. Our findings provide evidence on the long-term and accumulative cost of extreme temperature to middle-aged and elderly human capital, contributing to the understanding of the social cost of climate change and the consequent health inequality.