StarFlow: Generating Structured Workflow Outputs From Sketch Images
Journal:
arXiv
Published Date:
Mar 27, 2025
Abstract
Workflows are a fundamental component of automation in enterprise platforms,
enabling the orchestration of tasks, data processing, and system integrations.
Despite being widely used, building workflows can be complex, often requiring
manual configuration through low-code platforms or visual programming tools. To
simplify this process, we explore the use of generative foundation models,
particularly vision-language models (VLMs), to automatically generate
structured workflows from visual inputs. Translating hand-drawn sketches or
computer-generated diagrams into executable workflows is challenging due to the
ambiguity of free-form drawings, variations in diagram styles, and the
difficulty of inferring execution logic from visual elements. To address this,
we introduce StarFlow, a framework for generating structured workflow outputs
from sketches using vision-language models. We curate a diverse dataset of
workflow diagrams -- including synthetic, manually annotated, and real-world
samples -- to enable robust training and evaluation. We finetune and benchmark
multiple vision-language models, conducting a series of ablation studies to
analyze the strengths and limitations of our approach. Our results show that
finetuning significantly enhances structured workflow generation, outperforming
large vision-language models on this task.
