UTFpatgen: Pattern generation without borders

J 2026

UTFpatgen: Pattern generation without borders

METELKA, Ondřej a Petr SOJKA

Základní údaje

Originální název

UTFpatgen: Pattern generation without borders

Autoři

METELKA, Ondřej a Petr SOJKA

Vydání

TUGboat: The Communications of the TeX Users Group, Portland, Oregon, USA, TUG, 2026, 0896-3207

Další údaje

Jazyk

angličtina

Typ výsledku

Článek v odborném periodiku

Obor

10200 1.2 Computer and information sciences

Stát vydavatele

Spojené státy

Utajení

není předmětem státního či obchodního tajemství

Odkazy

Journal paper page with fulltext and metadata

Označené pro přenos do RIV

Ano

Organizační jednotka

Fakulta informatiky

DOI

https://doi.org/10.47397/tb/47-2/tb146metelka-utfpatgen

Klíčová slova česky

UTFpatgen; dělení slov; slovníkový problém; efektivita; vzory dělení slov; Patgen; generování vzorů; dataset pro srovnávání; seznamy rozdělených slov; učení s učitelem; hyphenation benchmark

Klíčová slova anglicky

UTFpatgen; hyphenation; dictionary problem; effectiveness; hyphenation patterns; Patgen; pattern generation; benchmark dataset; hyphenated wordlists; supervised learning; hyphenation benchmark

Štítky

competing patterns, computer typesetting, free software, hyphenation, hyphenation patterns, patgen, patterns, počítačová sazba, Typography and digital documents

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 4. 9. 2026 15:15, doc. RNDr. Petr Sojka, Ph.D.

Anotace

V originále

Originally developed to power TeX’s hyphenation engine, hyphenation patterns generated by Frank Liang’s PatGen algorithm have transcended their original typesetting boundaries to become the universal backbone for automated text hyphenation across typesetting engines, web browsers, DTP software, and digital readers. Building upon such widespread and impactful deployment, UTFpatgen, a new addition to the TeXLive program suite, modernizes this foundational technology by removing legacy 8-bit limitations. It enables the seamless generation of compact, highly accurate hyphenation patterns for UTF-8 text---from complex scripts with large grapheme inventories (such as polytonic Greek and Tamil) to intricate conjunct characters and multilingual pattern sets. UTFpatgen has been programmed as a drop-in replacement for PatGen. Although PatGen was designed 44 years ago and the interface shows its age, we felt it was preferable not to disturb the many existing workflows built around it. Written from scratch in the CWEB literate programming system, UTFpatgen replaces fixed-capacity static arrays with dynamic memory allocation to eliminate constraints on dictionary sizes and hyphenation levels. Furthermore, by introducing a variable-length byte-indexing schema, it provides fully scalable support for UTF-8-encoded input alphabets while maintaining strict backward compatibility at the interface level. We present empirical results from stress testing UTFpatgen against the hyph-bench evaluation benchmark, offering a direct comparison of the time and memory efficiency of both programs. Ultimately, UTFpatgen provides the TeX ecosystem with a flexible, modern alternative capable of supporting complex script workflows, structured pattern sets for multiple languages and dialects, and new pattern uses such as ligature suppression, weighted hyphenation, and other uses of generated hierarchical patterns, without borders such as limitations on data structure sizes, allocation, and the number of generated levels of patterns.

Návaznosti

MUNI/A/1873/2025, interní kód MU

Název: Komplexní data a systémy s využitím umělé inteligence

Investor: Masarykova univerzita, Komplexní data a systémy s využitím umělé inteligence

Citovat

METELKA, Ondřej a Petr SOJKA. UTFpatgen: Pattern generation without borders. TUGboat: The Communications of the TeX Users Group. Portland, Oregon, USA: TUG, 2026, roč. 47, č. 2, s. 258-263. ISSN 0896-3207. Dostupné z: https://doi.org/10.47397/tb/47-2/tb146metelka-utfpatgen.

@article{2586560,
   author = {Metelka, Ondřej and Sojka, Petr},
   article_location = {Portland, Oregon, USA},
   article_number = {2},
   doi = {https://doi.org/10.47397/tb/47-2/tb146metelka-utfpatgen},
   keywords = {UTFpatgen; hyphenation; dictionary problem; effectiveness; hyphenation patterns; Patgen; pattern generation; benchmark dataset; hyphenated wordlists; supervised learning; hyphenation benchmark},
   language = {eng},
   issn = {0896-3207},
   journal = {TUGboat: The Communications of the TeX Users Group},
   title = {UTFpatgen: Pattern generation without borders},
   url = {https://doi.org/10.47397/tb/47-2/tb146metelka-utfpatgen},
   volume = {47},
   year = {2026}
}

TY  - JOUR
ID  - 2586560
AU  - Metelka, Ondřej - Sojka, Petr
PY  - 2026
TI  - UTFpatgen: Pattern generation without borders
JF  - TUGboat: The Communications of the TeX Users Group
VL  - 47
IS  - 2
SP  - 258-263
EP  - 258-263
PB  - TUG
SN  - 08963207
KW  - UTFpatgen
KW  - hyphenation
KW  - dictionary problem
KW  - effectiveness
KW  - hyphenation patterns
KW  - Patgen
KW  - pattern generation
KW  - benchmark dataset
KW  - hyphenated wordlists
KW  - supervised learning
KW  - hyphenation benchmark
UR  - https://doi.org/10.47397/tb/47-2/tb146metelka-utfpatgen
N2  - Originally developed to power TeX’s hyphenation engine, hyphenation patterns generated by Frank Liang’s PatGen algorithm have transcended their original typesetting boundaries to become the universal backbone for automated text hyphenation across typesetting engines, web browsers, DTP software, and digital readers. Building upon such widespread and impactful deployment, UTFpatgen, a new addition to the TeXLive program suite, modernizes this foundational technology by removing legacy 8-bit limitations. It enables the seamless generation of compact, highly accurate hyphenation patterns for UTF-8 text---from complex scripts with large grapheme inventories (such as polytonic Greek and Tamil) to intricate conjunct characters and multilingual pattern sets. UTFpatgen has been programmed as a drop-in replacement for PatGen. Although PatGen was designed 44 years ago and the interface shows its age, we felt it was preferable not to disturb the many existing workflows built around it. Written from scratch in the CWEB literate programming system, UTFpatgen replaces fixed-capacity static arrays with dynamic memory allocation to eliminate constraints on dictionary sizes and hyphenation levels. Furthermore, by introducing a variable-length byte-indexing schema, it provides fully scalable support for UTF-8-encoded input alphabets while maintaining strict backward compatibility at the interface level. We present empirical results from stress testing UTFpatgen against the hyph-bench evaluation benchmark, offering a direct comparison of the time and memory efficiency of both programs. Ultimately, UTFpatgen provides the TeX ecosystem with a flexible, modern alternative capable of supporting complex script workflows, structured pattern sets for multiple languages and dialects, and new pattern uses such as ligature suppression, weighted hyphenation, and other uses of generated hierarchical patterns, without borders such as limitations on data structure sizes, allocation, and the number of generated levels of patterns.
ER  -

METELKA, Ondřej a Petr SOJKA. UTFpatgen: Pattern generation without borders. \textit{TUGboat: The Communications of the TeX Users Group}. Portland, Oregon, USA: TUG, 2026, roč.~47, č.~2, s.~258-263. ISSN~0896-3207. Dostupné z: https://doi.org/10.47397/tb/47-2/tb146metelka-utfpatgen.

Přehled o publikaci