Methods for Estimating and Improving Robustness of Language
Models.

D 2022

Methods for Estimating and Improving Robustness of Language Models.

ŠTEFÁNIK, Michal

Základní údaje

Originální název

Methods for Estimating and Improving Robustness of Language Models.

Autoři

ŠTEFÁNIK, Michal (703 Slovensko, garant, domácí)

Vydání

Seattle, Washington + Online, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, od s. 44-51, 8 s. 2022

Nakladatel

Association for Computational Linguistics

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Spojené státy

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

elektronická verze "online"

Odkazy

URL

Kód RIV

RIV/00216224:14330/22:00126309

Organizační jednotka

Fakulta informatiky

ISBN

978-1-7138-5621-4

DOI

http://dx.doi.org/10.18653/v1/2022.naacl-srw.6

UT WoS

000860760300006

Klíčová slova anglicky

natural language processing; transformers; robustness; generalization

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 6. 4. 2023 12:36, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

Despite their outstanding performance, large language models (LLMs) suffer notorious flaws related to their preference for shallow textual relations over full semantic complexity of the problem. This proposal investigates a common denominator of this problem in their weak ability to generalise outside of the training domain. We survey diverse research directions providing estimations of model generalisation ability and find that incorporating some of these measures in the training objectives leads to enhanced distributional robustness of neural models. Based on these findings, we present future research directions enhancing the robustness of LLMs.

Návaznosti

MUNI/A/1195/2021, interní kód MU

Název: Aplikovaný výzkum v oblastech vyhledávání, analýz a vizualizací rozsáhlých dat, zpracování přirozeného jazyka a aplikované umělé inteligence

Investor: Masarykova univerzita, Aplikovaný výzkum v oblastech vyhledávání, analýz a vizualizací rozsáhlých dat, zpracování přirozeného jazyka a aplikované umělé inteligence

Citovat

ŠTEFÁNIK, Michal. Methods for Estimating and Improving Robustness of Language Models. Online. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop. Seattle, Washington + Online: Association for Computational Linguistics, 2022, s. 44-51. ISBN 978-1-7138-5621-4. Dostupné z: https://dx.doi.org/10.18653/v1/2022.naacl-srw.6.

@inproceedings{1875957,
   author = {Štefánik, Michal},
   address = {Seattle, Washington + Online},
   booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop},
   doi = {http://dx.doi.org/10.18653/v1/2022.naacl-srw.6},
   keywords = {natural language processing; transformers; robustness; generalization},
   howpublished = {elektronická verze "online"},
   language = {eng},
   location = {Seattle, Washington + Online},
   isbn = {978-1-7138-5621-4},
   pages = {44-51},
   publisher = {Association for Computational Linguistics},
   title = {Methods for Estimating and Improving Robustness of Language Models.},
   url = {https://aclanthology.org/2022.naacl-srw.6/},
   year = {2022}
}

TY  - JOUR
ID  - 1875957
AU  - Štefánik, Michal
PY  - 2022
TI  - Methods for Estimating and Improving Robustness of Language Models.
PB  - Association for Computational Linguistics
CY  - Seattle, Washington + Online
SN  - 9781713856214
KW  - natural language processing
KW  - transformers
KW  - robustness
KW  - generalization
UR  - https://aclanthology.org/2022.naacl-srw.6/
N2  - Despite their outstanding performance, large language models (LLMs) suffer notorious flaws related to their preference for shallow textual relations over full semantic complexity of the problem. This proposal investigates a common denominator of this problem in their weak ability to generalise outside of the training domain. We survey diverse research directions providing estimations of model generalisation ability and find that incorporating some of these measures in the training objectives leads to enhanced distributional robustness of neural models. Based on these findings, we present future research directions enhancing the robustness of LLMs.
ER  -

ŠTEFÁNIK, Michal. Methods for Estimating and Improving Robustness of Language Models. Online. In \textit{Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop}. Seattle, Washington + Online: Association for Computational Linguistics, 2022, s.~44-51. ISBN~978-1-7138-5621-4. Dostupné z: https://dx.doi.org/10.18653/v1/2022.naacl-srw.6.

Podrobný výpis o publikaci