Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini

Journal: arXiv
Published Date:

Abstract

We introduce Gemini Embedding 2, a native multimodal embedding model that allows embedding video, audio, image, and text modalities in a unified representation space. We leverage the multimodal capabilities of Gemini to produce embeddings for arbitrary combinations of interleaved inputs across all these modalities that generalize well across a wide variety of tasks. Applying large-scale contrastive learning in a multi-task multi-stage training setup, we achieve state-of-the-art performance on key embedding benchmarks including unimodal, cross-modal, and multimodal retrieval spanning a diverse set of tasks. We show that our embedding model demonstrates strong performance (with a score of 62.9 R@1 on MSCOCO, 68.8 NDCG@10 on Vatex, 69.9 on MTEB multilingual and 84.0 on MTEB Code) across a variety of tasks surpassing the performance of specialized models. These unified capabilities make Gemini Embedding 2 a promising candidate for downstream use cases such as RAG, recommendation and search. Furthermore, its robust zero-shot performance across distinct fields - from astronomy and bioscience to fine arts and the culinary arts - establishes it as a highly reliable, out-of-the-box representation even for specialized domains.

Authors

  • Madhuri Shanbhogue; Zhe Li; Shanfeng Zhang; Gustavo Hernández Ábrego; Shih-Cheng Huang; Aashi Jain; Daniel Salz; Sonam Goenka; Chaitra Hegde; Ji Ma; Feiyang Chen; Jiaxing Wu; Tanmaya Dabral; Babak Samari; Kevin Poulet; Daniel Cer; Kaifeng Chen; Paul Suganathan; Hui Hui; Jovan Andonov; Philippe Schlattner; Jay Han; Iftekhar Naim; Wing Lowe; Vladimir Pchelin; Albert Yang; Yi-Ting Chen; Zhongli Ding; Grace Zhang; Georg Heigold; Yichang Chen; Antoine Reveillon; Brendan Mccloskey; Wenlei Zhou; Dahun Kim; Rui Meng; Emma Wang; Jack Zheng; Halley Fede; Zhen Yang; Keegan Mosley; Brian Potetz; Sahil Dua; Henrique Schechter Vera; Shen Gao; Hesen Zhang; Andreas Hess; Hengxuan Ying; Alberto Montes; Karan Gill; Min Choi; Sebastian Russo; Anja Hauth; Jinhyuk Lee; Michael Boratko; Megan Barnes; Vikram Rao; Claudiu Musat; Cyril Allauzen; Ehsan Variani; Shankar Kumar; Tom Bagby; Junyi Jiao; Yang Gu; Tengxin Li; Ayush Agrawal; Roberto Santana; Dev Nath; Stephen Karukas; Shuoxuan Han; Lucia Loher; Alice Twu; Nidhi Vyas; Siddharth Bhai; Frank Palma Gomez; Wangyuan Zhang; Chaoren Liu; Jizheng Yang; Steve Qiu; Shijie Zhang; Sujay Kulkarni; Sascha Rothe; Sean Nakamoto; Raphael Hoffmann; Zach Gleicher; Yunhsuan Sung; Qin Yin; Tom Duerig; Mojtaba Seyedhosseini