2024
Self-Training Language Models in Arithmetic Reasoning
KADLČÍK, Marek, Michal ŠTEFÁNIK, Ondřej SOTOLÁŘ a Vlastimil MARTINEKZákladní údaje
Originální název
Self-Training Language Models in Arithmetic Reasoning
Autoři
Vydání
ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024
Další údaje
Jazyk
angličtina
Typ výsledku
Prezentace na konferencích
Obor
10201 Computer sciences, information science, bioinformatics
Stát vydavatele
Česká republika
Utajení
není předmětem státního či obchodního tajemství
Odkazy
Organizační jednotka
Fakulta informatiky
Klíčová slova anglicky
language models, arithmetical reasoning, self-training, preference optimisation
Změněno: 12. 6. 2024 14:45, Mgr. Michal Štefánik
Anotace
V originále
Recent works show the impressive effectiveness of an agent framework in solving problems with language models. In this work, we apply two key features from the framework, interaction with tools and goal-oriented training, to improve models' arithmetical reasoning. First, we curate and transform existing datasets to create Calc-X, a standardized collection with over 300,000 problems with step-by-step solutions. We use Calc-X to train models we call Calcformers that interact with a calculator during inference. Calcformers achieve twice the accuracy of standard baselines. Finally, we optimize Calcformers via self-training using preference optimization and supervised loss by checking the model's predicted results. We find that self-training can achieve substantial improvements on out-of-domain problems and that traditional supervised loss is a strong baseline for preference optimization. Our results show that preference optimization converges faster and isn't prone to forgetting pre-trained abilities.
Návaznosti
| MUNI/A/1590/2023, interní kód MU |
|