CFG-Bench

Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents

Paper Code

Introduction

Multimodal Large Language Models (MLLMs) show promising results as decision-making engines for embodied agents operating in complex, physical environments. However, existing benchmarks often prioritize high-level planning or spatial reasoning, leaving the fine-grained action intelligence required for embodied physical interaction underexplored. To address this gap, we introduce CFG-Bench, a new benchmark designed to systematically evaluate this crucial capability. CFG-Bench consists of 1,368 curated videos paired with 19,562 three-modalities question-answer pairs targeting four cognitive abilities: 1) Physical Interaction, 2) Temporal-Causal Relation, 3) Intentional Understanding, and 4) Evaluative Judgment. Together, these dimensions provide a systematic framework for assessing a model's ability to translate visual observations into actionable knowledge, moving beyond mere surface-level recognition. Our comprehensive evaluation on CFG-Bench reveals that leading MLLMs struggle to produce detailed instructions for physical interactions and exhibit profound limitations in the higher-order reasoning of intention and evaluation. Moreover, supervised fine-tuning (SFT) on our data demonstrates that teaching an MLLMs to articulate fine-grained actions directly translates to significant performance gains on established embodied benchmarks. Our analysis highlights these limitations and offers insights for developing more capable and grounded embodied agents.

Motivation

Illustration of CFG-Bench's focus on embodied intelligence over descriptive accuracy. The top part shows how FAVOR-Bench annotates and questions from a third-person perspective, a task which current MLLMs can often solve. In contrast, the bottom part demonstrates CFG-Bench's fine-grained annotation and first-person scenario questions, which probes for the actionable physical and intentional details necessary for embodied agents. Current MLLMs struggle to master the crucial fine-grained details required for physical interaction.

Task Demonstration

Task demonstration of CFG-Bench. Note: all QA pairs, including those above, are slightly simplified for clarity and brevity.

Statistics

Data statistics of CFG-Bench. (a) Distribution and video length statistics of the five datasets. (b) The distribution of tasks across four tiers. AW means average words of questions.

Benchmark Comparison

data-composition

Comparison with other benchmarks. CFG-Bench introduces a four-tiered cognitive framework for embodied fine-grained intelligence, distinguishing it from existing action benchmarks focused on third-person coarse description and other embodied benchmarks that prioritize spatial reasoning and high-level planning over the fine-grained physical action. It also provides a more comprehensive evaluation protocol by uniquely integrating three-modalities QAs. ✓ indicates partial coverage.

Experimental Results

Results on CFG-Bench

grade-lv

Comprehensive results of leading MLLMs on CFG-Bench. It presents the performance of each task within every cognitive tier, along with the average score for open-ended questions Avgo and the average accuracy for close-ended questions Avgc. Random selection and human performance are also included for comparison. The highest and suboptimal results are bolded and underlined.

Fine-tuning with CFG Captions

Base Model (Qwen2.5-VL-7B)

The robotic arm approaches the glass and the bottle.
The robotic arm picks up the bottle.
The robotic arm tilts the bottle and pours the liquid into the glass.

SFT Model

The robotic hand is positioned on the left side of the frame. It slowly moves toward the glass bottle on the table. The robotic hand then grips the bottle's handle with its fingers, lifting it slightly off the table. As the bottle is lifted, the robotic hand rotates it counterclockwise to align the spout with the glass below. The hand then tilts the bottle and pours the liquid into the glass.

Base Model (Qwen2.5-VL-7B)

Person opens a bottle of vanilla extract with their right hand.
Person scoops vanilla extract with a spoon using their right hand.
Person pours the vanilla extract into a cup using their right hand.

SFT Model

The fingers of both hands are opening a bag placed on the table. During the process, they slowly lift and rotate the bag once. Then, the left hand holds the bag while the right hand uses the fingers to open its mouth. The right hand takes a spoon from the cup on the table, lifts it up, then inserts it into the bag, slowly scoops out some black powder, pours the powder into the cup.

Question-Answer Examples

Citation

        @article{liu2025beyond,
                title={Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents},
                author={Liu, Dayong and Xu, Chao and Chen, Weihong and Zhang, Suyu and Wang, Juncheng and Deng, Jiankang and Sun, Baigui and Liu, Yang},
                journal={arXiv preprint arXiv:2511.18685},
                year={2025}
                }