Estimators of average treatment effects under unconfounded treatment assignment are known to become rather imprecise if there is limited overlap in the covariate distributions between the treatment groups. But such limited overlap can also have a detrimental effect on inference, and lead for example to highly distorted confidence intervals. This paper shows that this is because the coverage error of traditional confidence intervals is not so much driven by the total sample size, but by the number of observations in the areas of limited overlap. At least some of these "local sample sizes" are often very small in applications, up to the point where distributional approximation derived from the Central Limit Theorem become unreliable. Building on this observation, the paper proposes two new robust confidence intervals that are extensions of classical approaches to small sample inference. It shows that these approaches are easy to implement, and have superior theoretical and practical properties relative to standard methods in empirically relevant settings. They should thus be useful for practitioners.