One Size Does Not Fit All: Finding the Optimal Subword Sizes
for FastText Models across Languages

D 2021

One Size Does Not Fit All: Finding the Optimal Subword Sizes for FastText Models across Languages

NOVOTNÝ, Vít, Eniafe Festus AYETIRAN, Dalibor BAČOVSKÝ, Dávid LUPTÁK, Michal ŠTEFÁNIK et. al.

Základní údaje

Originální název

One Size Does Not Fit All: Finding the Optimal Subword Sizes for FastText Models across Languages

Autoři

NOVOTNÝ, Vít (203 Česká republika, garant, domácí), Eniafe Festus AYETIRAN (566 Nigérie, domácí), Dalibor BAČOVSKÝ (203 Česká republika, domácí), Dávid LUPTÁK (703 Slovensko, domácí), Michal ŠTEFÁNIK (703 Slovensko, domácí) a Petr SOJKA (203 Česká republika, domácí)

Vydání

Varna, Bulgaria, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), od s. 1068-1074, 7 s. 2021

Nakladatel

INCOMA Ltd.

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

60203 Linguistics

Stát vydavatele

Bulharsko

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

tištěná verze "print"

Odkazy

DOI preprint sborníku preprint

Kód RIV

RIV/00216224:14330/21:00122017

Organizační jednotka

Fakulta informatiky

ISBN

978-954-452-072-4

ISSN

DOI

http://dx.doi.org/10.26615/978-954-452-072-4_120

Klíčová slova česky

fastText; učení reprezentace; slovní analogie; optimalizace hyperparametrů; modelování jazyka; vzdálenost jazyků

Klíčová slova anglicky

fastText; representation learning; word analogy; hyperparameter optimization; language modeling; language distance

Štítky

firank_B, language modeling, machine learning, similarity search

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 23. 5. 2022 14:55, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

Unsupervised representation learning of words from large multilingual corpora is useful for downstream tasks such as word sense disambiguation, semantic text similarity, and information retrieval. The representation precision of log-bilinear fastText models is mostly due to their use of subword information. In previous work, the optimization of fastText's subword sizes has not been fully explored, and non-English fastText models were trained using subword sizes optimized for English and German word analogy tasks. In our work, we find the optimal subword sizes on the English, German, Czech, Italian, Spanish, French, Hindi, Turkish, and Russian word analogy tasks. We then propose a simple n-gram coverage model and we show that it predicts better-than-default subword sizes on the Spanish, French, Hindi, Turkish, and Russian word analogy tasks. We show that the optimization of fastText's subword sizes matters and results in a 14% improvement on the Czech word analogy task. We also show that expensive parameter optimization can be replaced by a simple n-gram coverage model that consistently improves the accuracy of fastText models on the word analogy tasks by up to 3% compared to the default subword sizes, and that it is within 1% accuracy of the optimal subword sizes.

Návaznosti

MUNI/A/1573/2020, interní kód MU

Název: Aplikovaný výzkum: vyhledávání, analýza a vizualizace rozsáhlých dat, zpracování přirozeného jazyka, umělá inteligence pro analýzu biomedicínských obrazů.

Investor: Masarykova univerzita, Aplikovaný výzkum: vyhledávání, analýza a vizualizace rozsáhlých dat, zpracování přirozeného jazyka, umělá inteligence pro analýzu biomedicínských obrazů.

Citovat

NOVOTNÝ, Vít, Eniafe Festus AYETIRAN, Dalibor BAČOVSKÝ, Dávid LUPTÁK, Michal ŠTEFÁNIK a Petr SOJKA. One Size Does Not Fit All: Finding the Optimal Subword Sizes for FastText Models across Languages. In Mitkov, Ruslan and Angelova, Galia. Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021). Varna, Bulgaria: INCOMA Ltd., 2021, s. 1068-1074. ISBN 978-954-452-072-4. Dostupné z: https://dx.doi.org/10.26615/978-954-452-072-4_120.

@inproceedings{1784399,
   author = {Novotný, Vít and Ayetiran, Eniafe Festus and Bačovský, Dalibor and Lupták, Dávid and Štefánik, Michal and Sojka, Petr},
   address = {Varna, Bulgaria},
   booktitle = {Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)},
   doi = {http://dx.doi.org/10.26615/978-954-452-072-4_120},
   editor = {Mitkov, Ruslan and Angelova, Galia},
   keywords = {fastText; representation learning; word analogy; hyperparameter optimization; language modeling; language distance},
   howpublished = {tištěná verze "print"},
   language = {eng},
   location = {Varna, Bulgaria},
   isbn = {978-954-452-072-4},
   pages = {1068-1074},
   publisher = {INCOMA Ltd.},
   title = {One Size Does Not Fit All: Finding the Optimal Subword Sizes for FastText Models across Languages},
   url = {https://doi.org/10.26615/978-954-452-072-4_120},
   year = {2021}
}

TY  - JOUR
ID  - 1784399
AU  - Novotný, Vít - Ayetiran, Eniafe Festus - Bačovský, Dalibor - Lupták, Dávid - Štefánik, Michal - Sojka, Petr
PY  - 2021
TI  - One Size Does Not Fit All: Finding the Optimal Subword Sizes for FastText Models across Languages
PB  - INCOMA Ltd.
CY  - Varna, Bulgaria
SN  - 9789544520724
KW  - fastText
KW  - representation learning
KW  - word analogy
KW  - hyperparameter optimization
KW  - language modeling
KW  - language distance
UR  - https://doi.org/10.26615/978-954-452-072-4_120
N2  - Unsupervised representation learning of words from large multilingual corpora is useful for downstream tasks such as word sense disambiguation, semantic text similarity, and information retrieval. The representation precision of log-bilinear fastText models is mostly due to their use of subword information. In previous work, the optimization of fastText's subword sizes has not been fully explored, and non-English fastText models were trained using subword sizes optimized for English and German word analogy tasks. In our work, we find the optimal subword sizes on the English, German, Czech, Italian, Spanish, French, Hindi, Turkish, and Russian word analogy tasks. We then propose a simple n-gram coverage model and we show that it predicts better-than-default subword sizes on the Spanish, French, Hindi, Turkish, and Russian word analogy tasks. We show that the optimization of fastText's subword sizes matters and results in a 14% improvement on the Czech word analogy task. We also show that expensive parameter optimization can be replaced by a simple n-gram coverage model that consistently improves the accuracy of fastText models on the word analogy tasks by up to 3% compared to the default subword sizes, and that it is within 1% accuracy of the optimal subword sizes.
ER  -

NOVOTNÝ, Vít, Eniafe Festus AYETIRAN, Dalibor BAČOVSKÝ, Dávid LUPTÁK, Michal ŠTEFÁNIK a Petr SOJKA. One Size Does Not Fit All: Finding the Optimal Subword Sizes for FastText Models across Languages. In Mitkov, Ruslan and Angelova, Galia. \textit{Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)}. Varna, Bulgaria: INCOMA Ltd., 2021, s.~1068-1074. ISBN~978-954-452-072-4. Dostupné z: https://dx.doi.org/10.26615/978-954-452-072-4\_{}120.

Podrobný výpis o publikaci