Other formats:
BibTeX
LaTeX
RIS
@proceedings{2403039,
author = {Kadlčík, Marek and Štefánik, Michal and Sotolář, Ondřej and Martinek, Vlastimil},
booktitle = {ICLR 2024 Workshop on Large Language Model (LLM) Agents},
keywords = {language models, arithmetical reasoning, self-training, preference optimisation},
language = {eng},
title = {Self-Training Language Models in Arithmetic Reasoning},
url = {https://openreview.net/forum?id=zBh79GuLNO},
year = {2024}
}
TY - CONF
ID - 2403039
AU - Kadlčík, Marek - Štefánik, Michal - Sotolář, Ondřej - Martinek, Vlastimil
PY - 2024
TI - Self-Training Language Models in Arithmetic Reasoning
KW - language models, arithmetical reasoning, self-training, preference optimisation
UR - https://openreview.net/forum?id=zBh79GuLNO
N2 - Recent works show the impressive effectiveness of an agent framework in solving problems with language models. In this work, we apply two key features from the framework, interaction with tools and goal-oriented training, to improve models' arithmetical reasoning. First, we curate and transform existing datasets to create Calc-X, a standardized collection with over 300,000 problems with step-by-step solutions. We use Calc-X to train models we call Calcformers that interact with a calculator during inference. Calcformers achieve twice the accuracy of standard baselines. Finally, we optimize Calcformers via self-training using preference optimization and supervised loss by checking the model's predicted results. We find that self-training can achieve substantial improvements on out-of-domain problems and that traditional supervised loss is a strong baseline for preference optimization. Our results show that preference optimization converges faster and isn't prone to forgetting pre-trained abilities.
ER -
KADLČÍK, Marek, Michal ŠTEFÁNIK, Ondřej SOTOLÁŘ and Vlastimil MARTINEK. Self-Training Language Models in Arithmetic Reasoning. In \textit{ICLR 2024 Workshop on Large Language Model (LLM) Agents}. 2024.
|
