The text compares first year university students’ compulsory English achievement tests written in two modes of administration, i.e. pencil and paper and computer-based tests (PPT and CBT). The central research question investigated is whether the usage of computer-based language testing is justified in such a context. It first maps the theoretical discourse and comments on selected fundamental aspects of language testing practices, significant for the comparison itself. A mixed methods design is employed, where quantitative test data is used to measure the equivalency of the two modes and qualitative self-report feedback form data explores the main advantages and disadvantages of these, as viewed by the students examined.