Other formats:
BibTeX
LaTeX
RIS
@inproceedings{1365039, author = {Kovář, Vojtěch}, address = {Brno}, booktitle = {Tenth Workshop on Recent Advances in Slavonic Natural Language Processing, RASLAN 2016}, editor = {Aleš Horák, Pavel Rychlý, Adam Rambousek}, keywords = {NLP; inter-annotator agreement; low inter-annotator agreement; evaluation; application; application-based evaluation; word sketch; thesaurus; terminology}, howpublished = {tištěná verze "print"}, language = {eng}, location = {Brno}, isbn = {978-80-263-1095-2}, pages = {127-134}, publisher = {Tribun EU}, title = {Evaluating Natural Language Processing Tasks with Low Inter-Annotator Agreement: The Case of Corpus Applications}, year = {2016} }
TY - JOUR ID - 1365039 AU - Kovář, Vojtěch PY - 2016 TI - Evaluating Natural Language Processing Tasks with Low Inter-Annotator Agreement: The Case of Corpus Applications PB - Tribun EU CY - Brno SN - 9788026310952 KW - NLP KW - inter-annotator agreement KW - low inter-annotator agreement KW - evaluation KW - application KW - application-based evaluation KW - word sketch KW - thesaurus KW - terminology N2 - In Low inter-annotator agreement = an ill-defined problem?, we have argued that tasks with low inter-annotator agreement are really common in natural language processing (NLP) and they deserve an appropriate attention. We have also outlined a preliminary solution for their evaluation. In On evaluation of natural language processing tasks: Is gold standard evaluation methodology a good solution? , we have agitated for extrinsic application-based evaluation of NLP tasks and against the gold standard methodology which is currently almost the only one really used in the NLP field. This paper brings a synthesis of these two: For three practical tasks, that normally have so low inter-annotator agreement that they are considered almost irrelevant to any scentific evaluation, we introduce an application-based evaluation scenario which illustrates that it is not only possible to evaluate them in a scientific way, but that this type of evaluation is much more telling than the gold standard way. ER -
KOVÁŘ, Vojtěch. Evaluating Natural Language Processing Tasks with Low Inter-Annotator Agreement: The Case of Corpus Applications. In Aleš Horák, Pavel Rychlý, Adam Rambousek. \textit{Tenth Workshop on Recent Advances in Slavonic Natural Language Processing, RASLAN 2016}. Brno: Tribun EU, 2016, p.~127-134. ISBN~978-80-263-1095-2.
|