ŠTEFÁNIK, Michal. On Eliminating Inductive Biases of Deep Language Models. In ALPS 2021. 2021.
Other formats:   BibTeX LaTeX RIS
Basic information
Original name On Eliminating Inductive Biases of Deep Language Models
Authors ŠTEFÁNIK, Michal.
Edition ALPS 2021, 2021.
Other information
Original language English
Type of outcome Presentations at conferences
Field of Study 10200 1.2 Computer and information sciences
Country of publisher Czech Republic
Confidentiality degree is not subject to a state or trade secret
WWW URL
Organization unit Faculty of Informatics
Keywords in English nlp;transformers;inductive bias;generalisation
Tags inductive bias, model generalisation, NLP, transformers
Tags International impact
Changed by Changed by: RNDr. Pavel Šmerk, Ph.D., učo 3880. Changed: 23/5/2022 11:04.
Abstract
This poster outlines problems of modern neural language models with out-of-domain performance. It suggests that this might be a consequence of narrow model specialization. In order to eliminate this flaw, it suggests two main directions of future work: 1. Introduction of evaluative metrics can identify out-of-domain generalization abilities, while 2. Objective approach adjusts the training objective to respect the desired generalization properties of the system.
Links
MUNI/A/1573/2020, interní kód MUName: Aplikovaný výzkum: vyhledávání, analýza a vizualizace rozsáhlých dat, zpracování přirozeného jazyka, umělá inteligence pro analýzu biomedicínských obrazů.
Investor: Masaryk University
PrintDisplayed: 14/10/2024 00:52