Concept-aware Data Construction Improves In-context Learning of
Language Models

k 2024

Concept-aware Data Construction Improves In-context Learning of Language Models

ŠTEFÁNIK, Michal; Marek KADLČÍK a Petr SOJKA

Základní údaje

Originální název

Concept-aware Data Construction Improves In-context Learning of Language Models

Autoři

ŠTEFÁNIK, Michal; Marek KADLČÍK a Petr SOJKA

Vydání

ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024

Další údaje

Jazyk

angličtina

Typ výsledku

Prezentace na konferencích

Obor

10302 Condensed matter physics

Stát vydavatele

Česká republika

Utajení

není předmětem státního či obchodního tajemství

Odkazy

Poster and slides, Openreview, preprint

Označené pro přenos do RIV

Ano

Kód RIV

RIV/00216224:14330/24:00136030

Organizační jednotka

Fakulta informatiky

DOI

https://doi.org/10.48550/arXiv.2403.09703

Klíčová slova česky

velké jazykové modely; LLM; učení v kontextu; učení konceptů; trénování z užitím konceptů; CoAT

Klíčová slova anglicky

language models; LLM; in-context learning; Concept-aware Training; CoAT

Štítky

LLM

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 7. 4. 2025 18:05, doc. RNDr. Petr Sojka, Ph.D.

Anotace

V originále

Many recent language models (LMs) of the Transformers family are capable of in-context learning (ICL), manifested in the LMs' ability to perform a new task solely from its description in a natural language input. Previous work curating these models assumes that ICL emerges from vast over-parametrization or the scale of multi-task training, but recent theoretical work attributes ICL emergence to training data properties, creating in-context learners with small, synthetic data. Inspired by these findings, we propose Concept-aware Training (CoAT), a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. We find that by using CoAT, pre-trained transformers can learn to better utilise new latent concepts from demonstrations and that such ability makes ICL more robust to functional deficiencies of the previous models. Finally, we show that concept-aware in-context learning improves ICL performance on a majority of new tasks compared to traditional instruction tuning, reaching performance comparable to the multitask learners using magnitudes of more training data.

Návaznosti

MUNI/A/1590/2023, interní kód MU

Název: Využití technik umělé inteligence pro zpracování dat, komplexní analýzy a vizualizaci rozsáhlých dat

Investor: Masarykova univerzita, Využití technik umělé inteligence pro zpracování dat, komplexní analýzy a vizualizaci rozsáhlých dat

Citovat

ŠTEFÁNIK, Michal; Marek KADLČÍK a Petr SOJKA. Concept-aware Data Construction Improves In-context Learning of Language Models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models. 2024. Dostupné z: https://doi.org/10.48550/arXiv.2403.09703.

@proceedings{2403040,
   author = {Štefánik, Michal and Kadlčík, Marek and Sojka, Petr},
   booktitle = {ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models},
   doi = {https://doi.org/10.48550/arXiv.2403.09703},
   keywords = {language models; LLM; in-context learning; Concept-aware Training; CoAT},
   language = {eng},
   title = {Concept-aware Data Construction Improves In-context Learning of Language Models},
   url = {https://openreview.net/forum?id=KwDG8YHZnP},
   year = {2024}
}

TY  - CONF
ID  - 2403040
AU  - Štefánik, Michal - Kadlčík, Marek - Sojka, Petr
PY  - 2024
TI  - Concept-aware Data Construction Improves In-context Learning of Language Models
KW  - language models
KW  - LLM
KW  - in-context learning
KW  - Concept-aware Training
KW  - CoAT
UR  - https://openreview.net/forum?id=KwDG8YHZnP
N2  - Many recent language models (LMs) of the Transformers family are capable of in-context learning (ICL), manifested in the LMs' ability to perform a new task solely from its description in a natural language input. Previous work curating these models assumes that ICL emerges from vast over-parametrization or the scale of multi-task training, but recent theoretical work attributes ICL emergence to training data properties, creating in-context learners with small, synthetic data. Inspired by these findings, we propose Concept-aware Training (CoAT), a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. We find that by using CoAT, pre-trained transformers can learn to better utilise new latent concepts from demonstrations and that such ability makes ICL more robust to functional deficiencies of the previous models. Finally, we show that concept-aware in-context learning improves ICL performance on a majority of new tasks compared to traditional instruction tuning, reaching performance comparable to the multitask learners using magnitudes of more training data.
ER  -

ŠTEFÁNIK, Michal; Marek KADLČÍK a Petr SOJKA. Concept-aware Data Construction Improves In-context Learning of Language Models. In \textit{ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models}. 2024. Dostupné z: https://doi.org/10.48550/arXiv.2403.09703.

Přehled o publikaci