Další formáty:
BibTeX
LaTeX
RIS
@inproceedings{1210699,
author = {Suchomel, Vít and Michelfeit, Jan and Pomikálek, Jan},
address = {Brno},
booktitle = {Eighth Workshop on Recent Advances in Slavonic Natural Language Processing},
keywords = {tokenisation; corpus tool},
howpublished = {tištěná verze "print"},
language = {eng},
location = {Brno},
pages = {71-75},
publisher = {Tribun EU},
title = {Text Tokenisation Using unitok},
url = {https://nlp.fi.muni.cz/raslan/2014/14.pdf},
year = {2014}
}
TY - JOUR
ID - 1210699
AU - Suchomel, Vít - Michelfeit, Jan - Pomikálek, Jan
PY - 2014
TI - Text Tokenisation Using unitok
PB - Tribun EU
CY - Brno
KW - tokenisation
KW - corpus tool
UR - https://nlp.fi.muni.cz/raslan/2014/14.pdf
N2 - This paper presents unitok, a tool for tokenisation of text in many languages. Although a simple idea – exploiting spaces in the text to separate tokens – works well most of the time, the rest of observed cases is quite complicated, language dependent and requires a special treatment. The paper covers the overall design of unitok as well as the way the tool deals with some language or web data specific tokenisation cases. The rule what to consider a token is briefly described. The tool is compared to two other tokenisers in terms of output token count and tokenising speed. unitok is publicly available under the GPL licence at http://corpus.tools.
ER -
SUCHOMEL, Vít, Jan MICHELFEIT a Jan POMIKÁLEK. Text Tokenisation Using unitok. In \textit{Eighth Workshop on Recent Advances in Slavonic Natural Language Processing}. Brno: Tribun EU, 2014, s.~71-75. ISSN~2336-4289.
|
