Calc-X and Calcformers: Empowering Arithmetical
Chain-of-Thought through Interaction with Symbolic Systems

D 2023

Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems

KADLČÍK, Marek, Michal ŠTEFÁNIK, Ondřej SOTOLÁŘ and Vlastimil MARTINEK

Basic information

Original name

Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems

Authors

KADLČÍK, Marek (203 Czech Republic, belonging to the institution), Michal ŠTEFÁNIK (703 Slovakia, guarantor, belonging to the institution), Ondřej SOTOLÁŘ (203 Czech Republic) and Vlastimil MARTINEK (203 Czech Republic, belonging to the institution)

Edition

Singapore, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Main track, p. 12101-12108, 8 pp. 2023

Publisher

Association for Computational Linguistics

Other information

Language

English

Type of outcome

Stať ve sborníku

Field of Study

10201 Computer sciences, information science, bioinformatics

Country of publisher

United States of America

Confidentiality degree

není předmětem státního či obchodního tajemství

Publication form

electronic version available online

References:

Manuscript in proceedings

RIV identification code

RIV/00216224:14330/23:00131954

Organization unit

Faculty of Informatics

ISBN

979-8-89176-060-8

DOI

http://dx.doi.org/10.18653/v1/2023.emnlp-main.742

Keywords in English

language models; dataset; arithmetic reasoning; multistep reasoning

Abstract

V originále

Despite outstanding performance on many generation tasks, language models are notoriously inclined to make factual errors in tasks requiring arithmetic reasoning. To enable language models to circumvent this deficiency and offload critical computation to a symbolic system, we create a collection of Calc-X datasets that demonstrates the appropriate use of a calculator in reasoning chains. We survey and unify several existing chain-of-thoughts datasets into a proposed novel format, resulting in a standard collection of over 300,000 samples requiring arithmetic reasoning. Finally, we use the new collection to train open-source calculator-assisted language models and show that models trained on Calc-X almost double the accuracy of generating correct results compared to baselines. We make all Calc-X datasets and models publicly available.

Links

MUNI/A/1339/2022, interní kód MU

Name: Rozvoj technik pro zpracování dat pro podporu vyhledávání, analýz a vizualizací rozsáhlých datových souborů s využitím umělé inteligence

Investor: Masaryk University, Development of data processing techniques to support search, analysis and visualization of large datasets using artificial intelligence

Citovat

KADLČÍK, Marek, Michal ŠTEFÁNIK, Ondřej SOTOLÁŘ and Vlastimil MARTINEK. Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems. Online. In Houda Bouamor, Juan Pino, Kalika Bali. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Main track. Singapore: Association for Computational Linguistics, 2023, p. 12101-12108. ISBN 979-8-89176-060-8. Available from: https://dx.doi.org/10.18653/v1/2023.emnlp-main.742.

@inproceedings{2327138,
   author = {Kadlčík, Marek and Štefánik, Michal and Sotolář, Ondřej and Martinek, Vlastimil},
   address = {Singapore},
   booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Main track},
   doi = {http://dx.doi.org/10.18653/v1/2023.emnlp-main.742},
   editor = {Houda Bouamor, Juan Pino, Kalika Bali},
   keywords = {language models; dataset; arithmetic reasoning; multistep reasoning},
   howpublished = {elektronická verze "online"},
   language = {eng},
   location = {Singapore},
   isbn = {979-8-89176-060-8},
   pages = {12101-12108},
   publisher = {Association for Computational Linguistics},
   title = {Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems},
   url = {https://aclanthology.org/2023.emnlp-main.742},
   year = {2023}
}

TY  - JOUR
ID  - 2327138
AU  - Kadlčík, Marek - Štefánik, Michal - Sotolář, Ondřej - Martinek, Vlastimil
PY  - 2023
TI  - Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems
PB  - Association for Computational Linguistics
CY  - Singapore
SN  - 9798891760608
KW  - language models
KW  - dataset
KW  - arithmetic reasoning
KW  - multistep reasoning
UR  - https://aclanthology.org/2023.emnlp-main.742
N2  - Despite outstanding performance on many generation tasks, language models are notoriously inclined to make factual errors in tasks requiring arithmetic reasoning. To enable language models to circumvent this deficiency and offload critical computation to a symbolic system, we create a collection of Calc-X datasets that demonstrates the appropriate use of a calculator in reasoning chains. We survey and unify several existing chain-of-thoughts datasets into a proposed novel format, resulting in a standard collection of over 300,000 samples requiring arithmetic reasoning. Finally, we use the new collection to train open-source calculator-assisted language models and show that models trained on Calc-X almost double the accuracy of generating correct results compared to baselines. We make all Calc-X datasets and models publicly available.
ER  -

KADLČÍK, Marek, Michal ŠTEFÁNIK, Ondřej SOTOLÁŘ and Vlastimil MARTINEK. Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems. Online. In Houda Bouamor, Juan Pino, Kalika Bali. \textit{Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Main track}. Singapore: Association for Computational Linguistics, 2023, p.~12101-12108. ISBN~979-8-89176-060-8. Available from: https://dx.doi.org/10.18653/v1/2023.emnlp-main.742.

Detailed Information on Publication Record