Calc-X and Calcformers: Empowering Arithmetical
Chain-of-Thought through Interaction with Symbolic Systems

D 2023

Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems

KADLČÍK, Marek, Michal ŠTEFÁNIK, Ondřej SOTOLÁŘ a Vlastimil MARTINEK

Základní údaje

Originální název

Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems

Autoři

KADLČÍK, Marek (203 Česká republika, domácí), Michal ŠTEFÁNIK (703 Slovensko, garant, domácí), Ondřej SOTOLÁŘ (203 Česká republika) a Vlastimil MARTINEK (203 Česká republika, domácí)

Vydání

Singapore, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Main track, od s. 12101-12108, 8 s. 2023

Nakladatel

Association for Computational Linguistics

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Spojené státy

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

elektronická verze "online"

Odkazy

Manuscript in proceedings

Kód RIV

RIV/00216224:14330/23:00131954

Organizační jednotka

Fakulta informatiky

ISBN

979-8-89176-060-8

DOI

http://dx.doi.org/10.18653/v1/2023.emnlp-main.742

Klíčová slova anglicky

language models; dataset; arithmetic reasoning; multistep reasoning

Štítky

core_A, firank_1

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 21. 5. 2024 08:54, Mgr. Michal Štefánik

Anotace

V originále

Despite outstanding performance on many generation tasks, language models are notoriously inclined to make factual errors in tasks requiring arithmetic reasoning. To enable language models to circumvent this deficiency and offload critical computation to a symbolic system, we create a collection of Calc-X datasets that demonstrates the appropriate use of a calculator in reasoning chains. We survey and unify several existing chain-of-thoughts datasets into a proposed novel format, resulting in a standard collection of over 300,000 samples requiring arithmetic reasoning. Finally, we use the new collection to train open-source calculator-assisted language models and show that models trained on Calc-X almost double the accuracy of generating correct results compared to baselines. We make all Calc-X datasets and models publicly available.

Návaznosti

MUNI/A/1339/2022, interní kód MU

Název: Rozvoj technik pro zpracování dat pro podporu vyhledávání, analýz a vizualizací rozsáhlých datových souborů s využitím umělé inteligence

Investor: Masarykova univerzita, Rozvoj technik pro zpracování dat pro podporu vyhledávání, analýz a vizualizací rozsáhlých datových souborů s využitím umělé inteligence

Citovat

KADLČÍK, Marek, Michal ŠTEFÁNIK, Ondřej SOTOLÁŘ a Vlastimil MARTINEK. Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems. Online. In Houda Bouamor, Juan Pino, Kalika Bali. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Main track. Singapore: Association for Computational Linguistics, 2023, s. 12101-12108. ISBN 979-8-89176-060-8. Dostupné z: https://dx.doi.org/10.18653/v1/2023.emnlp-main.742.

@inproceedings{2327138,
   author = {Kadlčík, Marek and Štefánik, Michal and Sotolář, Ondřej and Martinek, Vlastimil},
   address = {Singapore},
   booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Main track},
   doi = {http://dx.doi.org/10.18653/v1/2023.emnlp-main.742},
   editor = {Houda Bouamor, Juan Pino, Kalika Bali},
   keywords = {language models; dataset; arithmetic reasoning; multistep reasoning},
   howpublished = {elektronická verze "online"},
   language = {eng},
   location = {Singapore},
   isbn = {979-8-89176-060-8},
   pages = {12101-12108},
   publisher = {Association for Computational Linguistics},
   title = {Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems},
   url = {https://aclanthology.org/2023.emnlp-main.742},
   year = {2023}
}

TY  - JOUR
ID  - 2327138
AU  - Kadlčík, Marek - Štefánik, Michal - Sotolář, Ondřej - Martinek, Vlastimil
PY  - 2023
TI  - Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems
PB  - Association for Computational Linguistics
CY  - Singapore
SN  - 9798891760608
KW  - language models
KW  - dataset
KW  - arithmetic reasoning
KW  - multistep reasoning
UR  - https://aclanthology.org/2023.emnlp-main.742
N2  - Despite outstanding performance on many generation tasks, language models are notoriously inclined to make factual errors in tasks requiring arithmetic reasoning. To enable language models to circumvent this deficiency and offload critical computation to a symbolic system, we create a collection of Calc-X datasets that demonstrates the appropriate use of a calculator in reasoning chains. We survey and unify several existing chain-of-thoughts datasets into a proposed novel format, resulting in a standard collection of over 300,000 samples requiring arithmetic reasoning. Finally, we use the new collection to train open-source calculator-assisted language models and show that models trained on Calc-X almost double the accuracy of generating correct results compared to baselines. We make all Calc-X datasets and models publicly available.
ER  -

KADLČÍK, Marek, Michal ŠTEFÁNIK, Ondřej SOTOLÁŘ a Vlastimil MARTINEK. Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems. Online. In Houda Bouamor, Juan Pino, Kalika Bali. \textit{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Main track}. Singapore: Association for Computational Linguistics, 2023, s.~12101-12108. ISBN~979-8-89176-060-8. Dostupné z: https://dx.doi.org/10.18653/v1/2023.emnlp-main.742.

Podrobný výpis o publikaci