When FastText Pays Attention: Efficient Estimation of Word
Representations using Constrained Positional Weighting

J 2022

When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting

NOVOTNÝ, Vít, Michal ŠTEFÁNIK, Eniafe Festus AYETIRAN, Petr SOJKA, Radim ŘEHŮŘEK et. al.

Basic information

Original name

When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting

Authors

NOVOTNÝ, Vít (203 Czech Republic, guarantor, belonging to the institution), Michal ŠTEFÁNIK (703 Slovakia, belonging to the institution), Eniafe Festus AYETIRAN (566 Nigeria, belonging to the institution), Petr SOJKA (203 Czech Republic, belonging to the institution) and Radim ŘEHŮŘEK (203 Czech Republic)

Edition

Journal of Universal Computer Science, New York, USA, J.UCS Consortium, 2022, 0948-695X

Other information

Language

English

Type of outcome

Článek v odborném periodiku

Field of Study

10201 Computer sciences, information science, bioinformatics

Country of publisher

Czech Republic

Confidentiality degree

není předmětem státního či obchodního tajemství

References:

preprint DOI

Impact factor

Impact factor: 1.000

RIV identification code

RIV/00216224:14330/22:00124923

Organization unit

Faculty of Informatics

DOI

http://dx.doi.org/10.3897/jucs.69619

UT WoS

000767374300005

Keywords in English

Word embeddings; fastText; attention

Abstract

V originále

In 2018, Mikolov et al. introduced the positional language model, which has characteristics of attention-based neural machine translation models and which achieved state-of-the-art performance on the intrinsic word analogy task. However, the positional model is not practically fast and it has never been evaluated on qualitative criteria or extrinsic tasks. We propose a constrained positional model, which adapts the sparse attention mechanism from neural machine translation to improve the speed of the positional model. We evaluate the positional and constrained positional models on three novel qualitative criteria and on language modeling. We show that the positional and constrained positional models contain interpretable information about the grammatical properties of words and outperform other shallow models on language modeling. We also show that our constrained model outperforms the positional model on language modeling and trains twice as fast.

Links

MUNI/A/1195/2021, interní kód MU

Name: Aplikovaný výzkum v oblastech vyhledávání, analýz a vizualizací rozsáhlých dat, zpracování přirozeného jazyka a aplikované umělé inteligence

Investor: Masaryk University

Citovat

NOVOTNÝ, Vít, Michal ŠTEFÁNIK, Eniafe Festus AYETIRAN, Petr SOJKA and Radim ŘEHŮŘEK. When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting. Journal of Universal Computer Science. New York, USA: J.UCS Consortium, 2022, vol. 28, No 2, p. 181-201. ISSN 0948-695X. Available from: https://dx.doi.org/10.3897/jucs.69619.

@article{1762317,
   author = {Novotný, Vít and Štefánik, Michal and Ayetiran, Eniafe Festus and Sojka, Petr and Řehůřek, Radim},
   article_location = {New York, USA},
   article_number = {2},
   doi = {http://dx.doi.org/10.3897/jucs.69619},
   keywords = {Word embeddings; fastText; attention},
   language = {eng},
   issn = {0948-695X},
   journal = {Journal of Universal Computer Science},
   title = {When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting},
   url = {https://arxiv.org/abs/2104.09691},
   volume = {28},
   year = {2022}
}

TY  - JOUR
ID  - 1762317
AU  - Novotný, Vít - Štefánik, Michal - Ayetiran, Eniafe Festus - Sojka, Petr - Řehůřek, Radim
PY  - 2022
TI  - When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting
JF  - Journal of Universal Computer Science
VL  - 28
IS  - 2
SP  - 181-201
EP  - 181-201
PB  - J.UCS Consortium
SN  - 0948695X
KW  - Word embeddings
KW  - fastText
KW  - attention
UR  - https://arxiv.org/abs/2104.09691
N2  - In 2018, Mikolov et al. introduced the positional language model, which has characteristics of attention-based neural machine translation models and which achieved state-of-the-art performance on the intrinsic word analogy task. However, the positional model is not practically fast and it has never been evaluated on qualitative criteria or extrinsic tasks. We propose a constrained positional model, which adapts the sparse attention mechanism from neural machine translation to improve the speed of the positional model. We evaluate the positional and constrained positional models on three novel qualitative criteria and on language modeling. We show that the positional and constrained positional models contain interpretable information about the grammatical properties of words and outperform other shallow models on language modeling. We also show that our constrained model outperforms the positional model on language modeling and trains twice as fast.
ER  -

NOVOTNÝ, Vít, Michal ŠTEFÁNIK, Eniafe Festus AYETIRAN, Petr SOJKA and Radim ŘEHŮŘEK. When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting. \textit{Journal of Universal Computer Science}. New York, USA: J.UCS Consortium, 2022, vol.~28, No~2, p.~181-201. ISSN~0948-695X. Available from: https://dx.doi.org/10.3897/jucs.69619.

Detailed Information on Publication Record