Bridging Text and Vision: A Multi-View Text-Vision Registration Approach for Cross-Modal Place Recognition
Journal:
arXiv
Published Date:
Feb 20, 2025
Abstract
Mobile robots necessitate advanced natural language understanding
capabilities to accurately identify locations and perform tasks such as package
delivery. However, traditional visual place recognition (VPR) methods rely
solely on single-view visual information and cannot interpret human language
descriptions. To overcome this challenge, we bridge text and vision by
proposing a multiview (360{\deg} views of the surroundings) text-vision
registration approach called Text4VPR for place recognition task, which is the
first method that exclusively utilizes textual descriptions to match a database
of images. Text4VPR employs the frozen T5 language model to extract global
textual embeddings. Additionally, it utilizes the Sinkhorn algorithm with
temperature coefficient to assign local tokens to their respective clusters,
thereby aggregating visual descriptors from images. During the training stage,
Text4VPR emphasizes the alignment between individual text-image pairs for
precise textual description. In the inference stage, Text4VPR uses the Cascaded
Cross-Attention Cosine Alignment (CCCA) to address the internal mismatch
between text and image groups. Subsequently, Text4VPR performs precisely place
match based on the descriptions of text-image groups. On Street360Loc, the
first text to image VPR dataset we created, Text4VPR builds a robust baseline,
achieving a leading top-1 accuracy of 57% and a leading top-10 accuracy of 92%
within a 5-meter radius on the test set, which indicates that localization from
textual descriptions to images is not only feasible but also holds significant
potential for further advancement, as shown in Figure 1.
