While language models excel in many tasks, they often fail to solve even trivial math problems. This work aims to enhance the arithmetical reasoning abilities of language models. This thesis will address the problem in three parts:

1. Dataset curation
The work will review existing datasets for arithmetical reasoning, organize them into a consistent format with explicit annotations of calculations and remove data leaks between train and test split with an n-gram lexical overlap method.

2. Training calculator-using models
The work will implement an inference method for language models that enables interaction with a calculator within generation, train calculator-using models of two sizes with supervised learning on the created dataset, and compare their accuracy to the traditional generation.

3. Self-training
Additionally, the work will develop a prototype of offline and online self-training to explore their potential for autonomously improving model accuracy. The offline experiment includes generating predictions on a subset of the training dataset using a trained model and applying selected preference-optimization methods to align the model toward generating solutions with a correct result. In the online setup, predictions will be generated dynamically during self-training.
Both experiments will compare the accuracy of preference-optimization methods to a self-training baseline with a supervised next-token cross-entropy loss.