CzeGPT-2 – Training New Model for Czech Generative Text Processing Evaluated with the Summarization Task

HÁJEK, Adam and Aleš HORÁK. CzeGPT-2 – Training New Model for Czech Generative Text Processing Evaluated with the Summarization Task. IEEE ACCESS. UNITED STATES: IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2024, vol. 2024, No 12, p. 34570-34581. ISSN 2169-3536. Available from: https://dx.doi.org/10.1109/ACCESS.2024.3371689.

Basic information

• Original name: CzeGPT-2 – Training New Model for Czech Generative Text Processing Evaluated with the Summarization Task
• Authors: HÁJEK, Adam (203 Czech Republic, belonging to the institution) and Aleš HORÁK (203 Czech Republic, guarantor, belonging to the institution).
• Edition: IEEE ACCESS, UNITED STATES, IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC, 2024, 2169-3536.

Other information

• Original language: English
• Type of outcome: Article in a journal
• Field of Study: 10200 1.2 Computer and information sciences
• Country of publisher: United States of America
• Confidentiality degree: is not subject to a state or trade secret
• WWW: URL
• Impact factor: Impact factor: 3.900 in 2022
• Organization unit: Faculty of Informatics
• Doi: http://dx.doi.org/10.1109/ACCESS.2024.3371689
• UT WoS: 001178339600001
• Keywords in English: Task analysis;Training;Measurement;Transformers;Decoding;Computational modeling;Vocabulary;Czech;GPT-2;large language model;model evaluation;model training;summarization
• Tags: International impact, Reviewed

Abstract

Automatic text summarization (ATS), alongside neural machine translation or question answering, is one of the leading tasks in Natural Language Processing (NLP). In recent years, ATS has experienced significant development, especially in the English NLP world. Modern approaches are mainly based on the versatile Transformer architecture proposed by Vaswani et al. in 2017, which has revolutionized the field, and was later tuned and adjusted to various needs of different tasks. Non-mainstream languages, with Czech taken as a representative, on the other hand, are a little bit behind these efforts and tend to use lighter or heuristic methods. With the new CzeGPT-2 model and abstractive summarizer, we would like to take a step forward detailing the process of training a GPT-2 generative transformer model for a new language with a comprehensive evaluation of the task of Czech summarization and pointing out the benefits of this approach. We also present an in-depth analysis of the errors in generated summaries, allowing to locate the model’s weak spots.},

Links

• LM2023062, research and development project: Name: Digitální výzkumná infrastruktura pro jazykové technologie, umění a humanitní vědy

Investor: Ministry of Education, Youth and Sports of the CR