Computer-based testing is increasingly being adopted by educational institutions worldwide. However, whether this transition from paper-based testing leads to different outcomes in student performance remains an open question. This paper assesses the impact of computer-based testing by examining a large-scale pilot programme for low-stakes exams implemented in Portugal in 2022. We leverage rich student-level data to implement pooled OLS and difference-in-differences approaches. Our results indicate that students who used computer-based testing performed worse than their peers using paper-based testing by 5 to 14 percentage points, on average. This negative effect is concentrated in specific question formats, namely questions requiring the analysis of figures. We discuss the implications of our findings for the large-scale implementation of computer-based testing.