The emergence of artificial intelligence (AI) tools has offered new ways of teaching and learning. Businesses have also highlighted the importance of AI literacy in the workplace as AI is transforming operations and entire industries. Given the importance of obtaining AI skills and the opportunities it provides, it is useful for students to gain experience interacting with the emerging technology. Yet, the optimal ways to incorporate AI in each class so that students’ knowledge acquisition in coursework does not suffer are still unclear. This paper examines the causal effect of AI-enhanced test preparation on student performance in an economics class. In a difference-in-differences framework, we compare the changes in students’ test scores after relative to before utilizing AI to enhance learning between students who completed a guided AI assignment to prepare and those who did not. The findings provide evidence that learning through AI does not necessarily improve students’ performance on formal exams. This does not mean that students should not learn how to use AI tools, but rather that they may not prepare for exams in all courses while simultaneously improving their AI skill set.