LLaPa: A Vision-Language Model Framework for Counterfactual-Aware Procedural Planning
Journal:
arXiv
Published Date:
Jul 11, 2025
Abstract
While large language models (LLMs) have advanced procedural planning for
embodied AI systems through strong reasoning abilities, the integration of
multimodal inputs and counterfactual reasoning remains underexplored. To tackle
these challenges, we introduce LLaPa, a vision-language model framework
designed for multimodal procedural planning. LLaPa generates executable action
sequences from textual task descriptions and visual environmental images using
vision-language models (VLMs). Furthermore, we enhance LLaPa with two auxiliary
modules to improve procedural planning. The first module, the Task-Environment
Reranker (TER), leverages task-oriented segmentation to create a task-sensitive
feature space, aligning textual descriptions with visual environments and
emphasizing critical regions for procedural execution. The second module, the
Counterfactual Activities Retriever (CAR), identifies and emphasizes potential
counterfactual conditions, enhancing the model's reasoning capability in
counterfactual scenarios. Extensive experiments on ActPlan-1K and ALFRED
benchmarks demonstrate that LLaPa generates higher-quality plans with superior
LCS and correctness, outperforming advanced models. The code and models are
available https://github.com/sunshibo1234/LLaPa.
