The goal of this thesis is to design and develop a benchmark to evaluate pragmatic understanding and reasoning in the Czech language using large language models (LLMs).

The thesis will:

1. Establish the theoretical foundations of cognitive pragmatics and provide a focused literature review of the linguistic aspects of Czech pragmatic phenomena, leading to a clearly defined scope and taxonomy of the phenomena to be evaluated.
2. Review LLM benchmarks relevant to the thesis topic.
3. Based on the theoretical and practical foundation, design benchmark tasks and the data format for the selected phenomena, including task definitions, input/output structure, dataset schema, labeling scheme, and difficulty controls.
4. Create the dataset, ensuring linguistic naturalness and controlling for ambiguity. Implement quality assurance through review and consistency checks. The expected dataset size will be in the thousands of varied items and phenomena, in accordance with the analyzed design.
5. Deliver a reproducible benchmark and evaluation pipeline. Define metrics, set a human baseline, and run evaluations across multiple LLMs.

The main deliverable is a reproducible benchmark that enables consistent comparison of multiple LLMs' Czech-language pragmatic competence.