HARMONY: A Scalable Distributed Vector Database for High-Throughput Approximate Nearest Neighbor Search
Journal:
arXiv
Published Date:
Jun 17, 2025
Abstract
Approximate Nearest Neighbor Search (ANNS) is essential for various
data-intensive applications, including recommendation systems, image retrieval,
and machine learning. Scaling ANNS to handle billions of high-dimensional
vectors on a single machine presents significant challenges in memory capacity
and processing efficiency. To address these challenges, distributed vector
databases leverage multiple nodes for the parallel storage and processing of
vectors. However, existing solutions often suffer from load imbalance and high
communication overhead, primarily due to traditional partition strategies that
fail to effectively distribute the workload. In this paper, we introduce
Harmony, a distributed ANNS system that employs a novel multi-granularity
partition strategy, combining dimension-based and vector-based partition. This
strategy ensures a balanced distribution of computational load across all nodes
while effectively minimizing communication costs. Furthermore, Harmony
incorporates an early-stop pruning mechanism that leverages the monotonicity of
distance computations in dimension-based partition, resulting in significant
reductions in both computational and communication overhead. We conducted
extensive experiments on diverse real-world datasets, demonstrating that
Harmony outperforms leading distributed vector databases, achieving 4.63 times
throughput on average in four nodes and 58% performance improvement over
traditional distribution for skewed workloads.
