Self-training Language Models for Arithmetic Reasoning

D 2024

Self-training Language Models for Arithmetic Reasoning

KADLČÍK, Marek a Michal ŠTEFÁNIK

Základní údaje

Originální název

Self-training Language Models for Arithmetic Reasoning

Autoři

KADLČÍK, Marek (203 Česká republika, domácí) a Michal ŠTEFÁNIK (703 Slovensko, garant, domácí)

Vydání

Hybrid, Miami, 2024 Conference on Empirical Methods in Natural Language Processing, Findings of EMNLP 2024, od s. 12378-12386, 9 s. 2024

Nakladatel

Association for Computational Linguistics

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Spojené státy

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

elektronická verze "online"

Kód RIV

RIV/00216224:14330/24:00137409

Organizační jednotka

Fakulta informatiky

ISBN

979-8-89176-168-1

DOI

https://doi.org/10.18653/v1/2024.findings-emnlp.721

EID Scopus

2-s2.0-85217622765

Klíčová slova anglicky

language models; arithmetic reasoning; self-training; implicit feedback; preference optimization

Štítky

firank_B

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 4. 4. 2025 01:13, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

Recent language models achieve impressive results in tasks involving complex multistep reasoning, but scaling these capabilities further traditionally requires expensive collection of more annotated data. In this work, we explore the potential of improving models' reasoning capabilities without new data, merely using automated feedback to the validity of their predictions in arithmetic reasoning (self-training). In systematic experimentation across six different arithmetic reasoning datasets, we find that models can substantially improve in both single-round (offline) and online self-training, reaching a correct result in +13.9% and +25.9% more cases, respectively, underlining the importance of actuality of self-training feedback. We further find that in the single-round, offline self-training, traditional supervised training can deliver gains comparable to preference optimization, but in online self-training, preference optimization methods largely outperform supervised training thanks to their superior stability and robustness on unseen types of problems.

Návaznosti

MUNI/A/1590/2023, interní kód MU

Název: Využití technik umělé inteligence pro zpracování dat, komplexní analýzy a vizualizaci rozsáhlých dat

Investor: Masarykova univerzita, Využití technik umělé inteligence pro zpracování dat, komplexní analýzy a vizualizaci rozsáhlých dat

MUNI/A/1608/2023, interní kód MU

Název: Zapojení studentů Fakulty informatiky do mezinárodní vědecké komunity 24

Investor: Masarykova univerzita, Zapojení studentů Fakulty informatiky do mezinárodní vědecké komunity 24

Citovat

KADLČÍK, Marek a Michal ŠTEFÁNIK. Self-training Language Models for Arithmetic Reasoning. Online. In Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen. 2024 Conference on Empirical Methods in Natural Language Processing, Findings of EMNLP 2024. Hybrid, Miami: Association for Computational Linguistics, 2024, s. 12378-12386. ISBN 979-8-89176-168-1. Dostupné z: https://doi.org/10.18653/v1/2024.findings-emnlp.721.

@inproceedings{2446097,
   author = {Kadlčík, Marek and Štefánik, Michal},
   address = {Hybrid, Miami},
   booktitle = {2024 Conference on Empirical Methods in Natural Language Processing, Findings of EMNLP 2024},
   doi = {https://doi.org/10.18653/v1/2024.findings-emnlp.721},
   editor = {Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen},
   keywords = {language models; arithmetic reasoning; self-training; implicit feedback; preference optimization},
   howpublished = {elektronická verze "online"},
   language = {eng},
   location = {Hybrid, Miami},
   isbn = {979-8-89176-168-1},
   pages = {12378-12386},
   publisher = {Association for Computational Linguistics},
   title = {Self-training Language Models for Arithmetic Reasoning},
   year = {2024}
}

TY  - CONF
ID  - 2446097
AU  - Kadlčík, Marek - Štefánik, Michal
PY  - 2024
TI  - Self-training Language Models for Arithmetic Reasoning
PB  - Association for Computational Linguistics
CY  - Hybrid, Miami
SN  - 9798891761681
KW  - language models
KW  - arithmetic reasoning
KW  - self-training
KW  - implicit feedback
KW  - preference optimization
N2  - Recent language models achieve impressive results in tasks involving complex multistep reasoning, but scaling these capabilities further traditionally requires expensive collection of more annotated data. In this work, we explore the potential of improving models' reasoning capabilities without new data, merely using automated feedback to the validity of their predictions in arithmetic reasoning (self-training). In systematic experimentation across six different arithmetic reasoning datasets, we find that models can substantially improve in both single-round (offline) and online self-training, reaching a correct result in +13.9% and +25.9% more cases, respectively, underlining the importance of actuality of self-training feedback. We further find that in the single-round, offline self-training, traditional supervised training can deliver gains comparable to preference optimization, but in online self-training, preference optimization methods largely outperform supervised training thanks to their superior stability and robustness on unseen types of problems.
ER  -

KADLČÍK, Marek a Michal ŠTEFÁNIK. Self-training Language Models for Arithmetic Reasoning. Online. In Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen. \textit{2024 Conference on Empirical Methods in Natural Language Processing, Findings of EMNLP 2024}. Hybrid, Miami: Association for Computational Linguistics, 2024, s.~12378-12386. ISBN~979-8-89176-168-1. Dostupné z: https://doi.org/10.18653/v1/2024.findings-emnlp.721.

Přehled o publikaci