Self-training Language Models for Arithmetic Reasoning

D 2024

Self-training Language Models for Arithmetic Reasoning

KADLČÍK, Marek and Michal ŠTEFÁNIK

Basic information

Original name

Self-training Language Models for Arithmetic Reasoning

Authors

KADLČÍK, Marek (203 Czech Republic, belonging to the institution) and Michal ŠTEFÁNIK (703 Slovakia, guarantor, belonging to the institution)

Edition

Hybrid, Miami, 2024 Conference on Empirical Methods in Natural Language Processing, Findings of EMNLP 2024, p. 12378-12386, 9 pp. 2024

Publisher

Association for Computational Linguistics

Other information

Language

English

Type of outcome

Proceedings paper

Field of Study

10201 Computer sciences, information science, bioinformatics

Country of publisher

United States of America

Confidentiality degree

is not subject to a state or trade secret

Publication form

electronic version available online

RIV identification code

RIV/00216224:14330/24:00137409

Organization unit

Faculty of Informatics

ISBN

979-8-89176-168-1

DOI

https://doi.org/10.18653/v1/2024.findings-emnlp.721

EID Scopus

2-s2.0-85217622765

Keywords in English

language models; arithmetic reasoning; self-training; implicit feedback; preference optimization

Abstract

In the original language

Recent language models achieve impressive results in tasks involving complex multistep reasoning, but scaling these capabilities further traditionally requires expensive collection of more annotated data. In this work, we explore the potential of improving models' reasoning capabilities without new data, merely using automated feedback to the validity of their predictions in arithmetic reasoning (self-training). In systematic experimentation across six different arithmetic reasoning datasets, we find that models can substantially improve in both single-round (offline) and online self-training, reaching a correct result in +13.9% and +25.9% more cases, respectively, underlining the importance of actuality of self-training feedback. We further find that in the single-round, offline self-training, traditional supervised training can deliver gains comparable to preference optimization, but in online self-training, preference optimization methods largely outperform supervised training thanks to their superior stability and robustness on unseen types of problems.

Links

MUNI/A/1590/2023, interní kód MU

Name: Využití technik umělé inteligence pro zpracování dat, komplexní analýzy a vizualizaci rozsáhlých dat

Investor: Masaryk University, Using artificial intelligence techniques for data processing, complex analysis and visualization of large-scale data

MUNI/A/1608/2023, interní kód MU

Name: Zapojení studentů Fakulty informatiky do mezinárodní vědecké komunity 24

Investor: Masaryk University

Cite

KADLČÍK, Marek and Michal ŠTEFÁNIK. Self-training Language Models for Arithmetic Reasoning. Online. In Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen. 2024 Conference on Empirical Methods in Natural Language Processing, Findings of EMNLP 2024. Hybrid, Miami: Association for Computational Linguistics, 2024, p. 12378-12386. ISBN 979-8-89176-168-1. Available from: https://doi.org/10.18653/v1/2024.findings-emnlp.721.

@inproceedings{2446097,
   author = {Kadlčík, Marek and Štefánik, Michal},
   address = {Hybrid, Miami},
   booktitle = {2024 Conference on Empirical Methods in Natural Language Processing, Findings of EMNLP 2024},
   doi = {https://doi.org/10.18653/v1/2024.findings-emnlp.721},
   editor = {Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen},
   keywords = {language models; arithmetic reasoning; self-training; implicit feedback; preference optimization},
   howpublished = {elektronická verze "online"},
   language = {eng},
   location = {Hybrid, Miami},
   isbn = {979-8-89176-168-1},
   pages = {12378-12386},
   publisher = {Association for Computational Linguistics},
   title = {Self-training Language Models for Arithmetic Reasoning},
   year = {2024}
}

TY  - CONF
ID  - 2446097
AU  - Kadlčík, Marek - Štefánik, Michal
PY  - 2024
TI  - Self-training Language Models for Arithmetic Reasoning
PB  - Association for Computational Linguistics
CY  - Hybrid, Miami
SN  - 9798891761681
KW  - language models
KW  - arithmetic reasoning
KW  - self-training
KW  - implicit feedback
KW  - preference optimization
N2  - Recent language models achieve impressive results in tasks involving complex multistep reasoning, but scaling these capabilities further traditionally requires expensive collection of more annotated data. In this work, we explore the potential of improving models' reasoning capabilities without new data, merely using automated feedback to the validity of their predictions in arithmetic reasoning (self-training). In systematic experimentation across six different arithmetic reasoning datasets, we find that models can substantially improve in both single-round (offline) and online self-training, reaching a correct result in +13.9% and +25.9% more cases, respectively, underlining the importance of actuality of self-training feedback. We further find that in the single-round, offline self-training, traditional supervised training can deliver gains comparable to preference optimization, but in online self-training, preference optimization methods largely outperform supervised training thanks to their superior stability and robustness on unseen types of problems.
ER  -

KADLČÍK, Marek and Michal ŠTEFÁNIK. Self-training Language Models for Arithmetic Reasoning. Online. In Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen. \textit{2024 Conference on Empirical Methods in Natural Language Processing, Findings of EMNLP 2024}. Hybrid, Miami: Association for Computational Linguistics, 2024, p.~12378-12386. ISBN~979-8-89176-168-1. Available from: https://doi.org/10.18653/v1/2024.findings-emnlp.721.

Přehled o publikaci