On Eliminating Inductive Biases of Deep Language Models

ŠTEFÁNIK, Michal. On Eliminating Inductive Biases of Deep Language Models. In ALPS 2021. 2021.

Basic information

Original name

On Eliminating Inductive Biases of Deep Language Models

Authors

ŠTEFÁNIK, Michal

Edition

ALPS 2021, 2021

---

Other information

Language

English

Type of outcome

Prezentace na konferencích

Field of Study

10200 1.2 Computer and information sciences

Country of publisher

Czech Republic

Confidentiality degree

není předmětem státního či obchodního tajemství

References:

URL

Organization unit

Faculty of Informatics

Keywords in English

nlp;transformers;inductive bias;generalisation

Tags

inductive bias, model generalisation, NLP, transformers

Tags

International impact

---

Abstract

V originále

This poster outlines problems of modern neural language models with out-of-domain performance. It suggests that this might be a consequence of narrow model specialization. In order to eliminate this flaw, it suggests two main directions of future work: 1. Introduction of evaluative metrics can identify out-of-domain generalization abilities, while 2. Objective approach adjusts the training objective to respect the desired generalization properties of the system.

---

Links

MUNI/A/1573/2020, interní kód MU

Name: Aplikovaný výzkum: vyhledávání, analýza a vizualizace rozsáhlých dat, zpracování přirozeného jazyka, umělá inteligence pro analýzu biomedicínských obrazů. Investor: Masaryk University