Generative Compositor for Few-Shot Visual Information Extraction
Journal:
arXiv
Published Date:
Mar 21, 2025
Abstract
Visual Information Extraction (VIE), aiming at extracting structured
information from visually rich document images, plays a pivotal role in
document processing. Considering various layouts, semantic scopes, and
languages, VIE encompasses an extensive range of types, potentially numbering
in the thousands. However, many of these types suffer from a lack of training
data, which poses significant challenges. In this paper, we propose a novel
generative model, named Generative Compositor, to address the challenge of
few-shot VIE. The Generative Compositor is a hybrid pointer-generator network
that emulates the operations of a compositor by retrieving words from the
source text and assembling them based on the provided prompts. Furthermore,
three pre-training strategies are employed to enhance the model's perception of
spatial context information. Besides, a prompt-aware resampler is specially
designed to enable efficient matching by leveraging the entity-semantic prior
contained in prompts. The introduction of the prompt-based retrieval mechanism
and the pre-training strategies enable the model to acquire more effective
spatial and semantic clues with limited training samples. Experiments
demonstrate that the proposed method achieves highly competitive results in the
full-sample training, while notably outperforms the baseline in the 1-shot,
5-shot, and 10-shot settings.
