When FastText Pays Attention: Efficient Estimation of Word
Representations using Constrained Positional Weighting

J 2022

When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting

NOVOTNÝ, Vít, Michal ŠTEFÁNIK, Eniafe Festus AYETIRAN, Petr SOJKA, Radim ŘEHŮŘEK et. al.

Základní údaje

Originální název

When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting

Autoři

NOVOTNÝ, Vít (203 Česká republika, garant, domácí), Michal ŠTEFÁNIK (703 Slovensko, domácí), Eniafe Festus AYETIRAN (566 Nigérie, domácí), Petr SOJKA (203 Česká republika, domácí) a Radim ŘEHŮŘEK (203 Česká republika)

Vydání

Journal of Universal Computer Science, New York, USA, J.UCS Consortium, 2022, 0948-695X

Další údaje

Jazyk

angličtina

Typ výsledku

Článek v odborném periodiku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Česká republika

Utajení

není předmětem státního či obchodního tajemství

Odkazy

preprint DOI

Impakt faktor

Impact factor: 1.000

Kód RIV

RIV/00216224:14330/22:00124923

Organizační jednotka

Fakulta informatiky

DOI

http://dx.doi.org/10.3897/jucs.69619

UT WoS

000767374300005

Klíčová slova anglicky

Word embeddings; fastText; attention

Štítky

attention, language modeling, Positional embeddings, word embeddings

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 23. 9. 2024 10:14, doc. RNDr. Petr Sojka, Ph.D.

Anotace

V originále

In 2018, Mikolov et al. introduced the positional language model, which has characteristics of attention-based neural machine translation models and which achieved state-of-the-art performance on the intrinsic word analogy task. However, the positional model is not practically fast and it has never been evaluated on qualitative criteria or extrinsic tasks. We propose a constrained positional model, which adapts the sparse attention mechanism from neural machine translation to improve the speed of the positional model. We evaluate the positional and constrained positional models on three novel qualitative criteria and on language modeling. We show that the positional and constrained positional models contain interpretable information about the grammatical properties of words and outperform other shallow models on language modeling. We also show that our constrained model outperforms the positional model on language modeling and trains twice as fast.

Návaznosti

MUNI/A/1195/2021, interní kód MU

Název: Aplikovaný výzkum v oblastech vyhledávání, analýz a vizualizací rozsáhlých dat, zpracování přirozeného jazyka a aplikované umělé inteligence

Investor: Masarykova univerzita, Aplikovaný výzkum v oblastech vyhledávání, analýz a vizualizací rozsáhlých dat, zpracování přirozeného jazyka a aplikované umělé inteligence

Citovat

NOVOTNÝ, Vít, Michal ŠTEFÁNIK, Eniafe Festus AYETIRAN, Petr SOJKA a Radim ŘEHŮŘEK. When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting. Journal of Universal Computer Science. New York, USA: J.UCS Consortium, 2022, roč. 28, č. 2, s. 181-201. ISSN 0948-695X. Dostupné z: https://dx.doi.org/10.3897/jucs.69619.

@article{1762317,
   author = {Novotný, Vít and Štefánik, Michal and Ayetiran, Eniafe Festus and Sojka, Petr and Řehůřek, Radim},
   article_location = {New York, USA},
   article_number = {2},
   doi = {http://dx.doi.org/10.3897/jucs.69619},
   keywords = {Word embeddings; fastText; attention},
   language = {eng},
   issn = {0948-695X},
   journal = {Journal of Universal Computer Science},
   title = {When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting},
   url = {https://arxiv.org/abs/2104.09691},
   volume = {28},
   year = {2022}
}

TY  - JOUR
ID  - 1762317
AU  - Novotný, Vít - Štefánik, Michal - Ayetiran, Eniafe Festus - Sojka, Petr - Řehůřek, Radim
PY  - 2022
TI  - When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting
JF  - Journal of Universal Computer Science
VL  - 28
IS  - 2
SP  - 181-201
EP  - 181-201
PB  - J.UCS Consortium
SN  - 0948695X
KW  - Word embeddings
KW  - fastText
KW  - attention
UR  - https://arxiv.org/abs/2104.09691
N2  - In 2018, Mikolov et al. introduced the positional language model, which has characteristics of attention-based neural machine translation models and which achieved state-of-the-art performance on the intrinsic word analogy task. However, the positional model is not practically fast and it has never been evaluated on qualitative criteria or extrinsic tasks. We propose a constrained positional model, which adapts the sparse attention mechanism from neural machine translation to improve the speed of the positional model. We evaluate the positional and constrained positional models on three novel qualitative criteria and on language modeling. We show that the positional and constrained positional models contain interpretable information about the grammatical properties of words and outperform other shallow models on language modeling. We also show that our constrained model outperforms the positional model on language modeling and trains twice as fast.
ER  -

NOVOTNÝ, Vít, Michal ŠTEFÁNIK, Eniafe Festus AYETIRAN, Petr SOJKA a Radim ŘEHŮŘEK. When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting. \textit{Journal of Universal Computer Science}. New York, USA: J.UCS Consortium, 2022, roč.~28, č.~2, s.~181-201. ISSN~0948-695X. Dostupné z: https://dx.doi.org/10.3897/jucs.69619.

Podrobný výpis o publikaci