Retrieving Semantically Similar Decisions under Noisy
Institutional Labels: Robust Comparison of Embedding Methods

j 2025

Retrieving Semantically Similar Decisions under Noisy Institutional Labels: Robust Comparison of Embedding Methods

NOVOTNÁ, Tereza a Jakub HARAŠTA

Základní údaje

Originální název

Retrieving Semantically Similar Decisions under Noisy Institutional Labels: Robust Comparison of Embedding Methods

Autoři

NOVOTNÁ, Tereza a Jakub HARAŠTA

Vydání

arXiv, arXiv:2512.05681, 2025

Další údaje

Jazyk

angličtina

Typ výsledku

Publikace v odborném periodiku – kromě recenzovaných typů article, review a letter

Obor

50501 Law

Stát vydavatele

Spojené státy

Utajení

není předmětem státního či obchodního tajemství

Odkazy

Text (arXiv.org)

Označené pro přenos do RIV

Organizační jednotka

Právnická fakulta

DOI

https://doi.org/10.48550/arXiv.2512.05681

Klíčová slova anglicky

legal information retrieval; case law; embeddings; evaluation under noisy labels; Czech Constitutional Court

Příznaky

Mezinárodní význam

Změněno: 27. 1. 2026 14:25, JUDr. Mgr. Jakub Harašta, Ph.D.

Anotace

V originále

Retrieving case law is a time-consuming task predominantly carried out by querying databases. We provide a comparison of two models in three different settings for Czech Constitutional Court decisions: (i) a large general-purpose embedder (OpenAI), (ii) a domain-specific BERT-trained from scratch on ~30,000 decisions using sliding windows and attention pooling. We propose a noise-aware evaluation including IDF-weighted keyword overlap as graded relevance, binarization via two thresholds (0.20 balanced, 0.28 strict), significance via paired bootstrap, and an nDCG diagnosis supported with qualitative analysis. Despite modest absolute nDCG (expected under noisy labels), the general OpenAI embedder decisively outperforms the domain pre-trained BERT in both settings at @10/@20/@100 across both thresholds; differences are statistically significant. Diagnostics attribute low absolutes to label drift and strong ideals rather than lack of utility. Additionally, our framework is robust enough to be used for evaluation under a noisy gold dataset, which is typical when handling data with heterogeneous labels stemming from legacy judicial databases.

Návaznosti

MPO 60273/24/21300/21000, interní kód MU

Název: CEDMO 2.0 NPO

Investor: Ministerstvo průmyslu a obchodu ČR, CEDMO 2.0 NPO

MUNI/G/1142/2022, interní kód MU

Název: Forensic Support for Building Trust in Smart Software Ecosystems

Investor: Masarykova univerzita, Forensic Support for Building Trust in Smart Software Ecosystems, INTERDISCIPLINARY - Mezioborové výzkumné projekty

Citovat

NOVOTNÁ, Tereza a Jakub HARAŠTA. Retrieving Semantically Similar Decisions under Noisy Institutional Labels: Robust Comparison of Embedding Methods. arXiv. arXiv:2512.05681, 2025, 13 s. Dostupné z: https://doi.org/10.48550/arXiv.2512.05681.

@article{2539097,
   author = {Novotná, Tereza and Harašta, Jakub},
   doi = {https://doi.org/10.48550/arXiv.2512.05681},
   keywords = {legal information retrieval; case law; embeddings; evaluation under noisy labels; Czech Constitutional Court},
   language = {eng},
   journal = {arXiv},
   title = {Retrieving Semantically Similar Decisions under Noisy Institutional Labels: Robust Comparison of Embedding Methods},
   url = {https://arxiv.org/abs/2512.05681},
   year = {2025}
}

TY  - JFULL
ID  - 2539097
AU  - Novotná, Tereza - Harašta, Jakub
PY  - 2025
TI  - Retrieving Semantically Similar Decisions under Noisy Institutional Labels: Robust Comparison of Embedding Methods
JF  - arXiv
PB  - arXiv:2512.05681
KW  - legal information retrieval
KW  - case law
KW  - embeddings
KW  - evaluation under noisy labels
KW  - Czech Constitutional Court
UR  - https://arxiv.org/abs/2512.05681
N2  - Retrieving case law is a time-consuming task predominantly carried out by querying databases. We provide a comparison of two models in three different settings for Czech Constitutional Court decisions: (i) a large general-purpose embedder (OpenAI), (ii) a domain-specific BERT-trained from scratch on ~30,000 decisions using sliding windows and attention pooling. We propose a noise-aware evaluation including IDF-weighted keyword overlap as graded relevance, binarization via two thresholds (0.20 balanced, 0.28 strict), significance via paired bootstrap, and an nDCG diagnosis supported with qualitative analysis. Despite modest absolute nDCG (expected under noisy labels), the general OpenAI embedder decisively outperforms the domain pre-trained BERT in both settings at @10/@20/@100 across both thresholds; differences are statistically significant. Diagnostics attribute low absolutes to label drift and strong ideals rather than lack of utility. Additionally, our framework is robust enough to be used for evaluation under a noisy gold dataset, which is typical when handling data with heterogeneous labels stemming from legacy judicial databases.
ER  -

NOVOTNÁ, Tereza a Jakub HARAŠTA. Retrieving Semantically Similar Decisions under Noisy Institutional Labels: Robust Comparison of Embedding Methods. \textit{arXiv}. arXiv:2512.05681, 2025, 13 s. Dostupné z: https://doi.org/10.48550/arXiv.2512.05681.

Přehled o publikaci