Teacher quality is a key factor in improving student academic achievement. As such, educational policymakers strive to design systems to hire the most effective teachers. This paper examines the effects of a national policy reform in Colombia that established a merit-based teacher-hiring system intended to enhance teacher quality and improve student learning. Implemented in 2005 for all public schools, the policy ties teacher-hiring decisions to candidates' performance on an exam evaluating subject-specific knowledge and teaching aptitude. The implementation of the policy led to many experienced contract teachers being replaced by high exam-performing novice teachers. We find that though the policy sharply increased pre-college test scores of teachers, it also decreased the overall stock of teacher experience and led to sharp decreases in students' exam performance and educational attainment. Using a difference-in-differences strategy to compare the outcomes of students from public and private schools over two decades, we show that the hiring reform decreased students' performance on high school exit exams by 8 percent of a standard deviation, and reduced the likelihood that students enroll in and graduate from college by more than 10 percent. The results underscore that relying exclusively on specific ex ante measures of teacher quality to screen candidates, particularly at the expense of teacher experience, may unintentionally reduce students' learning gains.