Other formats:
BibTeX
LaTeX
RIS
@inproceedings{2428378,
author = {Procházka, David and Slanináková, Terézia and Čerňanský, Jozef and Oľha, Jaroslav and Antol, Matej and Dohnal, Vlastislav},
booktitle = {17th International Conference on Similarity Search and Applications (SISAP 2024)},
keywords = {learned metric index;high-dimensional data;memory efficiency;on-disk index;approximate nearest neighbor search;similarity search},
howpublished = {elektronická verze "online"},
language = {eng},
publisher = {Springer},
title = {Scaling Learned Metric Index to 100M Datasets},
year = {2024}
}
TY - JOUR
ID - 2428378
AU - Procházka, David - Slanináková, Terézia - Čerňanský, Jozef - Oľha, Jaroslav - Antol, Matej - Dohnal, Vlastislav
PY - 2024
TI - Scaling Learned Metric Index to 100M Datasets
PB - Springer
KW - learned metric index;high-dimensional data;memory efficiency;on-disk index;approximate nearest neighbor search;similarity search
N2 - Learned indexing of high-dimensional data is an indexing approach that is still in the process of proving its viability – the Learned Metric Index (LMI) stands as one of the pioneering methods in this regard. Earlier implementation of LMI [Slanináková et al., SISAP 2023] primarily served as experimental prototype, operating under unrealistic assumptions, such as the availability of unlimited main memory or unbounded index construction time. Recently, however, LMI made the leap towards practical applicability on real-world datasets when it was successfully deployed to efficiently index 214 million protein structures for near-instantaneous retrieval [Procházka et al., Nucleic Acids Research 2024]. This paper details the key improvements that enabled this transition, including the introduction of parallel query processing (with the possibility of GPU acceleration), adaptive memory usage, pre-construction of memory buckets for contiguous access, a shift from k-means to spherical k-means clustering, and faster index construction through fewer epochs and the use of smaller training samples. LMI is now capable of handling 100M datasets and supports both in-memory and on-disk indexing, marking several important steps towards practical viability of AI-enhanced indexes for high-dimensional complex data in real-world settings.
ER -
PROCHÁZKA, David, Terézia SLANINÁKOVÁ, Jozef ČERŇANSKÝ, Jaroslav OĽHA, Matej ANTOL and Vlastislav DOHNAL. Scaling Learned Metric Index to 100M Datasets. Online. In \textit{17th International Conference on Similarity Search and Applications (SISAP 2024)}. Springer, 2024, 8 pp.
|
