A new powerful language model called ALBERT was presented by Google in 2019 when it reached state-of-the-art results in various NLP tasks for English (question answering, language understanding). ALBERT is based on the idea of transfer learning, where it requires a huge text dataset for pretraining general language representations and in successive separate training it can be fine tuned for a specific task.

The goal of this thesis is to pretrain a ALBERT model(s) for the Czech language, evaluate the model efficiency in several extrinsic tasks and compare the results to other current state-of-the-art Czech NLP language models.

The submitted thesis will consist of a thesis text and a practical part. The textual part will include a detailed description of ALBERT and other related language modelling approaches and information about available pretrained models suitable for comparison, a description of the pretraining process, and the details of the evaluation and comparison. The practical part will contain all code and data for replicating the results with the respective licenses for publication where possible.