Pre-trained Language Models Learn Remarkably Accurate
Representations of Numbers

D 2025

Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers

KADLČÍK, Marek; Michal ŠTEFÁNIK; Timothee MICKUS; Michal SPIEGEL; Josef KUCHAŘ et al.

Základní údaje

Originální název

Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers

Autoři

KADLČÍK, Marek; Michal ŠTEFÁNIK; Timothee MICKUS; Michal SPIEGEL a Josef KUCHAŘ

Vydání

Suzhou, China, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, od s. 26693-26702, 10 s. 2025

Nakladatel

Association for Computational Linguistics

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

elektronická verze "online"

Odkazy

URL

Označené pro přenos do RIV

Ano

Organizační jednotka

Fakulta informatiky

ISBN

979-8-89176-332-6

Klíčová slova anglicky

robustness; model editing; interpretability; probing; language models

Štítky

core_A, firank_1

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 21. 11. 2025 11:42, Mgr. Marek Kadlčík

Anotace

V originále

Pretrained language models (LMs) are prone to arithmetic errors. Existing work showed limited success in probing numeric values from models’ representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. However, we observe that previous probing methods are inadequate for the emergent structure of learned number embeddings with sinusoidal patterns. In response, we propose a novel probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. This proves that after the sole pre-training, LMs represent numbers with remarkable precision. Finally, we find that the embeddings’ preciseness judged by our probe’s accuracy explains a large portion of LM’s errors in elementary arithmetic, and show that aligning the embeddings with the pattern discovered by our probe can mitigate these errors.

Návaznosti

MUNI/A/1666/2024, interní kód MU

Název: Zapojení studentů Fakulty informatiky do mezinárodní vědecké komunity 25

Investor: Masarykova univerzita, Zapojení studentů Fakulty informatiky do mezinárodní vědecké komunity 25

Citovat

KADLČÍK, Marek; Michal ŠTEFÁNIK; Timothee MICKUS; Michal SPIEGEL a Josef KUCHAŘ. Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers. Online. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics, 2025, s. 26693-26702. ISBN 979-8-89176-332-6.

@inproceedings{2534464,
   author = {Kadlčík, Marek and Štefánik, Michal and Mickus, Timothee and Spiegel, Michal and Kuchař, Josef},
   address = {Suzhou, China},
   booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
   keywords = {robustness; model editing; interpretability; probing; language models},
   howpublished = {elektronická verze "online"},
   language = {eng},
   location = {Suzhou, China},
   isbn = {979-8-89176-332-6},
   pages = {26693-26702},
   publisher = {Association for Computational Linguistics},
   title = {Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers},
   url = {https://aclanthology.org/2025.emnlp-main.1356/},
   year = {2025}
}

TY  - CONF
ID  - 2534464
AU  - Kadlčík, Marek - Štefánik, Michal - Mickus, Timothee - Spiegel, Michal - Kuchař, Josef
PY  - 2025
TI  - Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers
PB  - Association for Computational Linguistics
CY  - Suzhou, China
SN  - 9798891763326
KW  - robustness
KW  - model editing
KW  - interpretability
KW  - probing
KW  - language models
UR  - https://aclanthology.org/2025.emnlp-main.1356/
N2  - Pretrained language models (LMs) are prone to arithmetic errors. Existing work showed limited success in probing numeric values from models’ representations, indicating that these errors can be attributed to the inherent unreliability of distributionally learned embeddings in representing exact quantities. However, we observe that previous probing methods are inadequate for the emergent structure of learned number embeddings with sinusoidal patterns. In response, we propose a novel probing technique that decodes numeric values from input embeddings with near-perfect accuracy across a range of open-source LMs. This proves that after the sole pre-training, LMs represent numbers with remarkable precision. Finally, we find that the embeddings’ preciseness judged by our probe’s accuracy explains a large portion of LM’s errors in elementary arithmetic, and show that aligning the embeddings with the pattern discovered by our probe can mitigate these errors.
ER  -

KADLČÍK, Marek; Michal ŠTEFÁNIK; Timothee MICKUS; Michal SPIEGEL a Josef KUCHAŘ. Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers. Online. In \textit{Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing}. Suzhou, China: Association for Computational Linguistics, 2025, s.~26693-26702. ISBN~979-8-89176-332-6.

Přehled o publikaci