Can In-context Learners Learn a Reasoning Concept from
Demonstrations?

D 2023

Can In-context Learners Learn a Reasoning Concept from Demonstrations?

ŠTEFÁNIK, Michal and Marek KADLČÍK

Basic information

Original name

Can In-context Learners Learn a Reasoning Concept from Demonstrations?

Authors

ŠTEFÁNIK, Michal (703 Slovakia, guarantor, belonging to the institution) and Marek KADLČÍK (203 Czech Republic, belonging to the institution)

Edition

Toronto, Canada, Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), p. 107-115, 9 pp. 2023

Publisher

The Association for Computational Linguistics

Other information

Language

English

Type of outcome

Stať ve sborníku

Field of Study

10201 Computer sciences, information science, bioinformatics

Country of publisher

United States of America

Confidentiality degree

není předmětem státního či obchodního tajemství

Publication form

electronic version available online

RIV identification code

RIV/00216224:14330/23:00131346

Organization unit

Faculty of Informatics

ISBN

978-1-959429-94-4

ISSN

Keywords in English

in-context learning; few-shot learning; generalization

Změněno: 7/4/2024 23:08, RNDr. Pavel Šmerk, Ph.D.

Abstract

V originále

Language models exhibit an emergent ability to learn a new task from a small number of input-output demonstrations. However, recent work shows that in-context learners largely rely on their pre-trained knowledge, such as the sentiment of the labels, instead of learning new associations from the input. We argue that the commonly-used few-shot evaluation using a random selection of in-context demonstrations can not disentangle models' reliance on such biases, as most of the randomly-selected demonstrations do not present relations informative for prediction beyond exposing the task's input-output distribution. Therefore, to evaluate models' in-context learning ability independent of models' memory, we introduce a Concept-sharing few-shot learning method choosing the demonstrations that share an underlying concept with the predicted sample. We extract a set of such concepts from available human explanations and measure how much models can benefit from presenting these concepts in few-shot demonstrations. We find that most of the recent in-context learners can not consistently benefit from the demonstrated concepts, irrespective of the model size. However, we note that T0 models are more sensitive to exhibited concepts, benefiting from concept-sharing demonstrations in 7 out of 8 evaluation scenarios.

Links

MUNI/A/1339/2022, interní kód MU

Name: Rozvoj technik pro zpracování dat pro podporu vyhledávání, analýz a vizualizací rozsáhlých datových souborů s využitím umělé inteligence

Investor: Masaryk University, Development of data processing techniques to support search, analysis and visualization of large datasets using artificial intelligence

Citovat

ŠTEFÁNIK, Michal and Marek KADLČÍK. Can In-context Learners Learn a Reasoning Concept from Demonstrations?. Online. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE). Toronto, Canada: The Association for Computational Linguistics, 2023, p. 107-115. ISBN 978-1-959429-94-4.

@inproceedings{2300360,
   author = {Štefánik, Michal and Kadlčík, Marek},
   address = {Toronto, Canada},
   booktitle = {Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE)},
   keywords = {in-context learning; few-shot learning; generalization},
   howpublished = {elektronická verze "online"},
   language = {eng},
   location = {Toronto, Canada},
   isbn = {978-1-959429-94-4},
   pages = {107-115},
   publisher = {The Association for Computational Linguistics},
   title = {Can In-context Learners Learn a Reasoning Concept from Demonstrations?},
   year = {2023}
}

TY  - JOUR
ID  - 2300360
AU  - Štefánik, Michal - Kadlčík, Marek
PY  - 2023
TI  - Can In-context Learners Learn a Reasoning Concept from Demonstrations?
PB  - The Association for Computational Linguistics
CY  - Toronto, Canada
SN  - 9781959429944
KW  - in-context learning
KW  - few-shot learning
KW  - generalization
N2  - Language models exhibit an emergent ability to learn a new task from a small number of input-output demonstrations. However, recent work shows that in-context learners largely rely on their pre-trained knowledge, such as the sentiment of the labels, instead of learning new associations from the input. We argue that the commonly-used few-shot evaluation using a random selection of in-context demonstrations can not disentangle models' reliance on such biases, as most of the randomly-selected demonstrations do not present relations informative for prediction beyond exposing the task's input-output distribution. Therefore, to evaluate models' in-context learning ability independent of models' memory, we introduce a Concept-sharing few-shot learning method choosing the demonstrations that share an underlying concept with the predicted sample. We extract a set of such concepts from available human explanations and measure how much models can benefit from presenting these concepts in few-shot demonstrations. We find that most of the recent in-context learners can not consistently benefit from the demonstrated concepts, irrespective of the model size. However, we note that T0 models are more sensitive to exhibited concepts, benefiting from concept-sharing demonstrations in 7 out of 8 evaluation scenarios.
ER  -

ŠTEFÁNIK, Michal and Marek KADLČÍK. Can In-context Learners Learn a Reasoning Concept from Demonstrations?. Online. In \textit{Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE)}. Toronto, Canada: The Association for Computational Linguistics, 2023, p.~107-115. ISBN~978-1-959429-94-4.

Detailed Information on Publication Record