The student will describe, compare, and evaluate the capabilities and performance of Python libraries for manipulating and storing dense vector embeddings on disk. Specifically, the student will compare the NumPy, Pandas, h5py, PyTables, PyArrow, and SQLite libraries and their underlying formats, including Pickle, NPY, CSV, HDF5, Zarr, Apache Parquet, and Apache Arrow. Each library will be described in terms of at least the following features: the ability to efficiently retrieve an embedding based on its user-specified identifier, support for metadata and its indexing for efficient filtering, and iterative or chunk-based data access. Format properties such as data orientation and compression support will also be discussed.

A performance evaluation of each library will be conducted through a series of benchmarks, including inserting single and bulk embeddings, removing single and bulk embeddings, reading embeddings by their position and embedding identifier, reading multiple embeddings stored both consecutively and randomly across the dataset, appending new embeddings, and serializing and deserializing the entire dataset. In addition, the student will explore and experimentally evaluate selected file compression techniques to reduce the overall size of serialized embeddings on disk. The performance tests will be evaluated using a combination of HDD and SSD storage on a subset of embeddings from the LAION-5B image collection and the DEEP1B dataset from ANN-Benchmarks.