BenCzechMark : A Czech-Centric Multitask and Multimetric
Benchmark for Large Language Models with Duel Scoring Mechanism

J 2025

BenCzechMark : A Czech-Centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism

FAJCIK, Martin; Martin DOCEKAL; Jan DOLEZAL; Karel ONDREJ; Karel BENES et al.

Základní údaje

Originální název

BenCzechMark : A Czech-Centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism

Autoři

Vydání

TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, CAMBRIDGE, MIT PRESS, 2025, 2307-387X

Další údaje

Jazyk

angličtina

Typ výsledku

Článek v odborném periodiku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Spojené státy

Utajení

není předmětem státního či obchodního tajemství

Odkazy

URL

Impakt faktor

Impact factor: 6.900 v roce 2024

Označené pro přenos do RIV

Ano

Kód RIV

RIV/00216224:14330/25:00142533

Organizační jednotka

Fakulta informatiky

Klíčová slova anglicky

Large language models; Czech; Benchmark

Změněno: 2. 4. 2026 15:46, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

We present BenCzechMark (BCM), the first comprehensive Czech language benchmark designed for large language models, offering diverse tasks, multiple task formats, and multiple evaluation metrics. Its duel scoring system is grounded in statistical significance theory and uses aggregation across tasks inspired by social preference theory. Our benchmark encompasses 50 challenging tasks, with corresponding test datasets, primarily in native Czech, with 14 newly collected ones. These tasks span 8 categories and cover diverse domains, including historical Czech news, essays from pupils or language learners, and spoken word. Furthermore, we collect and clean BUT-Large Czech Collection, the largest publicly available clean Czech language corpus, and use it for (i) contamination analysis and (ii) continuous pretraining of the first Czech-centric 7B language model with Czech-specific tokenization. We use our model as a baseline for comparison with publicly available multilingual models. Lastly, we release and maintain a leaderboard with existing 50 model submissions, where new model submissions can be made at https://huggingface.co /spaces/CZLC/BenCzechMark.

Návaznosti

LM2023062, projekt VaV

Název: Digitální výzkumná infrastruktura pro jazykové technologie, umění a humanitní vědy

Investor: Ministerstvo školství, mládeže a tělovýchovy ČR, LINDAT/CLARIAH-CZ - Digitální výzkumná infrastruktura pro jazykové technologie, umění a humanitní vědy

OSCARS-01-247, interní kód MU

Název: Oscars - Opravidlo 2.0 – Public Online Proofreading Service

Investor: Evropská unie, Oscars - Opravidlo 2.0 – Public Online Proofreading Service, Výzkumné infrastruktury

90254, velká výzkumná infrastruktura

Název: e-INFRA CZ II

Citovat

FAJCIK, Martin; Martin DOCEKAL; Jan DOLEZAL; Karel ONDREJ; Karel BENES; Jan KAPSA; Pavel SMRZ; Alexander POLOK; Michal HRADIS; Zuzana NEVĚŘILOVÁ; Aleš HORÁK; Radoslav SABOL; Michal ŠTEFÁNIK; Adam JIRKOVSKY; David ADAMCZYK; Petr HYNER; Jan HULA a Hynek KYDLICEK. BenCzechMark : A Czech-Centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism. TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS. CAMBRIDGE: MIT PRESS, 2025, roč. 13, č. 1, s. 1068-1095. ISSN 2307-387X. Dostupné z: https://doi.org/10.1162/TACL.a.32.

@article{2532139,
   author = {Fajcik, Martin and Docekal, Martin and Dolezal, Jan and Ondrej, Karel and Benes, Karel and Kapsa, Jan and Smrz, Pavel and Polok, Alexander and Hradis, Michal and Nevěřilová, Zuzana and Horák, Aleš and Sabol, Radoslav and Štefánik, Michal and Jirkovsky, Adam and Adamczyk, David and Hyner, Petr and Hula, Jan and Kydlicek, Hynek},
   article_location = {CAMBRIDGE},
   article_number = {1},
   doi = {https://doi.org/10.1162/TACL.a.32},
   keywords = {Large language models; Czech; Benchmark},
   language = {eng},
   issn = {2307-387X},
   journal = {TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS},
   title = {BenCzechMark : A Czech-Centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism},
   url = {https://direct.mit.edu/tacl/article/doi/10.1162/TACL.a.32/132962/BenCzechMark-A-Czech-Centric-Multitask-and},
   volume = {13},
   year = {2025}
}

TY  - JOUR
ID  - 2532139
AU  - Fajcik, Martin - Docekal, Martin - Dolezal, Jan - Ondrej, Karel - Benes, Karel - Kapsa, Jan - Smrz, Pavel - Polok, Alexander - Hradis, Michal - Nevěřilová, Zuzana - Horák, Aleš - Sabol, Radoslav - Štefánik, Michal - Jirkovsky, Adam - Adamczyk, David - Hyner, Petr - Hula, Jan - Kydlicek, Hynek
PY  - 2025
TI  - BenCzechMark : A Czech-Centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism
JF  - TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS
VL  - 13
IS  - 1
SP  - 1068-1095
EP  - 1068-1095
PB  - MIT PRESS
SN  - 2307387X
KW  - Large language models
KW  - Czech
KW  - Benchmark
UR  - https://direct.mit.edu/tacl/article/doi/10.1162/TACL.a.32/132962/BenCzechMark-A-Czech-Centric-Multitask-and
N2  - We present BenCzechMark (BCM), the first comprehensive Czech language benchmark designed for large language models, offering diverse tasks, multiple task formats, and multiple evaluation metrics. Its duel scoring system is grounded in statistical significance theory and uses aggregation across tasks inspired by social preference theory. Our benchmark encompasses 50 challenging tasks, with corresponding test datasets, primarily in native Czech, with 14 newly collected ones. These tasks span 8 categories and cover diverse domains, including historical Czech news, essays from pupils or language learners, and spoken word. Furthermore, we collect and clean BUT-Large Czech Collection, the largest publicly available clean Czech language corpus, and use it for (i) contamination analysis and (ii) continuous pretraining of the first Czech-centric 7B language model with Czech-specific tokenization. We use our model as a baseline for comparison with publicly available multilingual models. Lastly, we release and maintain a leaderboard with existing 50 model submissions, where new model submissions can be made at https://huggingface.co /spaces/CZLC/BenCzechMark.
ER  -

FAJCIK, Martin; Martin DOCEKAL; Jan DOLEZAL; Karel ONDREJ; Karel BENES; Jan KAPSA; Pavel SMRZ; Alexander POLOK; Michal HRADIS; Zuzana NEVĚŘILOVÁ; Aleš HORÁK; Radoslav SABOL; Michal ŠTEFÁNIK; Adam JIRKOVSKY; David ADAMCZYK; Petr HYNER; Jan HULA a Hynek KYDLICEK. BenCzechMark : A Czech-Centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism. \textit{TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS}. CAMBRIDGE: MIT PRESS, 2025, roč.~13, č.~1, s.~1068-1095. ISSN~2307-387X. Dostupné z: https://doi.org/10.1162/TACL.a.32.

Přehled o publikaci