The Art of Reproducible Machine Learning: A Survey of
Methodology in Word Vector Experiments

D 2020

The Art of Reproducible Machine Learning: A Survey of Methodology in Word Vector Experiments

NOVOTNÝ, Vít

Basic information

Original name

The Art of Reproducible Machine Learning: A Survey of Methodology in Word Vector Experiments

Authors

NOVOTNÝ, Vít (203 Czech Republic, guarantor, belonging to the institution)

Edition

Brno, Proceedings of the Fourteenth Workshop on Recent Advances in Slavonic Natural Language Processing, RASLAN 2020, p. 55-64, 10 pp. 2020

Publisher

Tribun EU

Other information

Language

English

Type of outcome

Stať ve sborníku

Field of Study

10201 Computer sciences, information science, bioinformatics

Country of publisher

Czech Republic

Confidentiality degree

není předmětem státního či obchodního tajemství

Publication form

printed version "print"

References:

PDF Domovská stránka workshopu

RIV identification code

RIV/00216224:14330/20:00117106

Organization unit

Faculty of Informatics

ISBN

978-80-263-1600-8

ISSN

UT WoS

000655471300006

Keywords in English

Machine learning; word vectors; word2vec; fastText; word analogy; reproducibility

Abstract

V originále

Since the seminal work of Mikolov et al. (2013), word vectors of log-bilinear SVMs have found their way into many NLP applications as an unsupervised measure of word relatedness.

Due to the rapid pace of research and the publish-or-perish mantra of academic publishing, word vector experiments contain undisclosed parameters, which make them difficult to reproduce.

In our work, we introduce the experiments and their parameters, compare the published experimental results with our own, and suggest default parameter settings and ways to make previous and future experiments easier to reproduce.

We show that the lack of variable control can cause up to 24% difference in accuracy on the word analogy tasks.

Links

MUNI/A/1076/2019, interní kód MU

Name: Zapojení studentů Fakulty informatiky do mezinárodní vědecké komunity 20 (Acronym: SKOMU)

Investor: Masaryk University, Category A

MUNI/A/1411/2019, interní kód MU

Name: Aplikovaný výzkum: softwarové architektury kritických infrastruktur, bezpečnost počítačových systémů, zpracování přirozeného jazyka a jazykové inženýrství, vizualizaci velkých dat a rozšířená realita.

Investor: Masaryk University, Category A

Citovat

NOVOTNÝ, Vít. The Art of Reproducible Machine Learning: A Survey of Methodology in Word Vector Experiments. In Aleš Horák, Pavel Rychlý, Adam Rambousek. Proceedings of the Fourteenth Workshop on Recent Advances in Slavonic Natural Language Processing, RASLAN 2020. Brno: Tribun EU, 2020, p. 55-64. ISBN 978-80-263-1600-8.

@inproceedings{1699699,
   author = {Novotný, Vít},
   address = {Brno},
   booktitle = {Proceedings of the Fourteenth Workshop on Recent Advances in Slavonic Natural Language Processing, RASLAN 2020},
   editor = {Aleš Horák, Pavel Rychlý, Adam Rambousek},
   keywords = {Machine learning; word vectors; word2vec; fastText; word analogy; reproducibility},
   howpublished = {tištěná verze "print"},
   language = {eng},
   location = {Brno},
   isbn = {978-80-263-1600-8},
   pages = {55-64},
   publisher = {Tribun EU},
   title = {The Art of Reproducible Machine Learning: A Survey of Methodology in Word Vector Experiments},
   url = {https://nlp.fi.muni.cz/raslan/raslan20.pdf#page=63},
   year = {2020}
}

TY  - JOUR
ID  - 1699699
AU  - Novotný, Vít
PY  - 2020
TI  - The Art of Reproducible Machine Learning: A Survey of Methodology in Word Vector Experiments
PB  - Tribun EU
CY  - Brno
SN  - 9788026316008
KW  - Machine learning
KW  - word vectors
KW  - word2vec
KW  - fastText
KW  - word analogy
KW  - reproducibility
UR  - https://nlp.fi.muni.cz/raslan/raslan20.pdf#page=63
N2  -

Since the seminal work of Mikolov et al. (2013), word vectors of log-bilinear SVMs have found their way into many NLP applications as an unsupervised measure of word relatedness.

Due to the rapid pace of research and the publish-or-perish mantra of academic publishing, word vector experiments contain undisclosed parameters, which make them difficult to reproduce.

We show that the lack of variable control can cause up to 24% difference in accuracy on the word analogy tasks.

ER -

NOVOTNÝ, Vít. The Art of Reproducible Machine Learning: A Survey of Methodology in Word Vector Experiments. In Aleš Horák, Pavel Rychlý, Adam Rambousek. \textit{Proceedings of the Fourteenth Workshop on Recent Advances in Slavonic Natural Language Processing, RASLAN 2020}. Brno: Tribun EU, 2020, p.~55-64. ISBN~978-80-263-1600-8.

Detailed Information on Publication Record