The aim of the thesis is to design, develop, and implement a tool for token-level language identification based on a pre-trained language model, with a focus on multilingual texts and code-switching.

The theoretical part will provide an overview of current approaches to language identification, particularly at the token level, as well as a review of available datasets and tools for processing multilingual data.

The practical part will include the design of different methods for creating training data and a comparison of different types of pre-trained language models in terms of their suitability for the given task. The outcome will be an implemented tool, its experimental evaluation on real-world data, and an analysis of conditions for efficient deployment when processing large volumes of data. It will also include a comparison with existing language identification systems, evaluated at the sentence level. In addition, a web interface will be provided to demonstrate the tool’s functionality.