Concept-aware Data Construction Improves In-context Learning of
Language Models

D 2024

Concept-aware Data Construction Improves In-context Learning of Language Models

ŠTEFÁNIK, Michal, Marek KADLČÍK a Petr SOJKA

Základní údaje

Originální název

Concept-aware Data Construction Improves In-context Learning of Language Models

Autoři

ŠTEFÁNIK, Michal (703 Slovensko, garant, domácí), Marek KADLČÍK (203 Česká republika, domácí) a Petr SOJKA (203 Česká republika, domácí)

Vydání

Bangkok, Thailand, Findings of the Association for Computational Linguistics ACL 2024, od s. 12335-12352, 18 s. 2024

Nakladatel

Association for Computational Linguistics

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Spojené státy

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

elektronická verze "online"

Odkazy

fulltext

Organizační jednotka

Fakulta informatiky

ISBN

979-8-89176-099-8

Klíčová slova česky

efektivita dat; hluboké jazykové modely; učení s vědomím konceptů; kontextové učení

Klíčová slova anglicky

data efficiency; LLM; concept-aware training; in-context learning

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 24. 10. 2024 19:04, Mgr. Michal Štefánik

Anotace

V originále

Many recent language models (LMs) are capable of in-context learning (ICL), manifested in the LMs’ ability to perform a new task solely from natural-language instruction. Previous work curating in-context learners assumes that ICL emerges from a vast over-parametrization or the scale of multi-task training. However, recent theoretical work attributes the ICL ability to concept-dependent training data and creates functional in-context learners even in small-scale, synthetic settings.In this work, we practically explore this newly identified axis of ICL quality. We propose Concept-aware Training (CoAT), a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. We find that by using CoAT, pre-trained transformers can learn to better utilise new latent concepts from demonstrations and that such ability makes ICL more robust to the functional deficiencies of the previous models. Finally, we show that concept-aware in-context learners are much more effective in in-context learning a majority of unseen tasks compared to traditional instruction tuning, and fare comparably also to previous in-context learners trained in large-scale multitask learning requiring magnitudes of more training data.

Návaznosti

MUNI/A/1590/2023, interní kód MU

Název: Využití technik umělé inteligence pro zpracování dat, komplexní analýzy a vizualizaci rozsáhlých dat

Investor: Masarykova univerzita, Využití technik umělé inteligence pro zpracování dat, komplexní analýzy a vizualizaci rozsáhlých dat

MUNI/A/1608/2023, interní kód MU

Název: Zapojení studentů Fakulty informatiky do mezinárodní vědecké komunity 24

Investor: Masarykova univerzita, Zapojení studentů Fakulty informatiky do mezinárodní vědecké komunity 24

Citovat

ŠTEFÁNIK, Michal, Marek KADLČÍK a Petr SOJKA. Concept-aware Data Construction Improves In-context Learning of Language Models. Online. In Lun-Wei Ku, Andre Martins, Vivek Srikumar. Findings of the Association for Computational Linguistics ACL 2024. Bangkok, Thailand: Association for Computational Linguistics, 2024, s. 12335-12352. ISBN 979-8-89176-099-8.

@inproceedings{2425558,
   author = {Štefánik, Michal and Kadlčík, Marek and Sojka, Petr},
   address = {Bangkok, Thailand},
   booktitle = {Findings of the Association for Computational Linguistics ACL 2024},
   editor = {Lun-Wei Ku, Andre Martins, Vivek Srikumar},
   keywords = {data efficiency; LLM; concept-aware training; in-context learning},
   howpublished = {elektronická verze "online"},
   language = {eng},
   location = {Bangkok, Thailand},
   isbn = {979-8-89176-099-8},
   pages = {12335-12352},
   publisher = {Association for Computational Linguistics},
   title = {Concept-aware Data Construction Improves In-context Learning of Language Models},
   url = {https://aclanthology.org/2024.findings-acl.733/},
   year = {2024}
}

TY  - JOUR
ID  - 2425558
AU  - Štefánik, Michal - Kadlčík, Marek - Sojka, Petr
PY  - 2024
TI  - Concept-aware Data Construction Improves In-context Learning of Language Models
PB  - Association for Computational Linguistics
CY  - Bangkok, Thailand
SN  - 9798891760998
KW  - data efficiency
KW  - LLM
KW  - concept-aware training
KW  - in-context learning
UR  - https://aclanthology.org/2024.findings-acl.733/
N2  - Many recent language models (LMs) are capable of in-context learning (ICL), manifested in the LMs’ ability to perform a new task solely from natural-language instruction. Previous work curating in-context learners assumes that ICL emerges from a vast over-parametrization or the scale of multi-task training. However, recent theoretical work attributes the ICL ability to concept-dependent training data and creates functional in-context learners even in small-scale, synthetic settings.In this work, we practically explore this newly identified axis of ICL quality. We propose Concept-aware Training (CoAT), a framework for constructing training scenarios that make it beneficial for the LM to learn to utilize the analogical reasoning concepts from demonstrations. We find that by using CoAT, pre-trained transformers can learn to better utilise new latent concepts from demonstrations and that such ability makes ICL more robust to the functional deficiencies of the previous models. Finally, we show that concept-aware in-context learners are much more effective in in-context learning a majority of unseen tasks compared to traditional instruction tuning, and fare comparably also to previous in-context learners trained in large-scale multitask learning requiring magnitudes of more training data.
ER  -

ŠTEFÁNIK, Michal, Marek KADLČÍK a Petr SOJKA. Concept-aware Data Construction Improves In-context Learning of Language Models. Online. In Lun-Wei Ku, Andre Martins, Vivek Srikumar. \textit{Findings of the Association for Computational Linguistics ACL 2024}. Bangkok, Thailand: Association for Computational Linguistics, 2024, s.~12335-12352. ISBN~979-8-89176-099-8.

Podrobný výpis o publikaci