SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches
Journal:
arXiv
Published Date:
Feb 11, 2025
Abstract
Text-to-image models can generate visually appealing images from text
descriptions. Efforts have been devoted to improving model controls with prompt
tuning and spatial conditioning. However, our formative study highlights the
challenges for non-expert users in crafting appropriate prompts and specifying
fine-grained spatial conditions (e.g., depth or canny references) to generate
semantically cohesive images, especially when multiple objects are involved. In
response, we introduce SketchFlex, an interactive system designed to improve
the flexibility of spatially conditioned image generation using rough region
sketches. The system automatically infers user prompts with rational
descriptions within a semantic space enriched by crowd-sourced object
attributes and relationships. Additionally, SketchFlex refines users' rough
sketches into canny-based shape anchors, ensuring the generation quality and
alignment of user intentions. Experimental results demonstrate that SketchFlex
achieves more cohesive image generations than end-to-end models, meanwhile
significantly reducing cognitive load and better matching user intentions
compared to region-based generation baseline.
