OneDB: A Distributed Multi-Metric Data Similarity Search System
Journal:
arXiv
Published Date:
Jul 6, 2025
Abstract
Increasingly massive volumes of multi-modal data are being accumulated in
many {real world} settings, including in health care and e-commerce. This
development calls for effective general-purpose data management solutions for
multi-modal data. Such a solution must facilitate user-friendly and accurate
retrieval of any multi-modal data according to diverse application
requirements. Further, such a solution must be capable of efficient and
scalable retrieval.
To address this need, we present OneDB, a distributed multi-metric data
similarity retrieval system. This system exploits the fact that data of diverse
modalities, such as text, images, and video, can be represented as metric data.
The system thus affords each data modality its own metric space with its own
distance function and then uses a multi-metric model to unify multi-modal data.
The system features several innovations: (i) an extended Spart SQL query
interface; (ii) lightweight means of learning appropriate weights of different
modalities when retrieving multi-modal data to enable accurate retrieval; (iii)
smart search-space pruning strategies that improve efficiency; (iv) two-layered
indexing of data to ensure load-balancing during distributed processing; and
(v) end-to-end system parameter autotuning.
Experiments on three real-life datasets and two synthetic datasets offer
evidence that the system is capable of state-of-the-art performance: (i)
efficient and effective weight learning; (ii) retrieval accuracy improvements
of 12.63\%--30.75\% over the state-of-the-art vector similarity search system
at comparable efficiency; (iii) accelerated search by 2.5--5.75x over
state-of-the-art single- or multi-metric solutions; (iv) demonstrated high
scalability; and (v) parameter tuning that enables performance improvements of
15+%.
