Title: Semantic-Geometric Task Representations for Bimanual Manipulation from Human Demonstrations to Robot Action Planning

Abstract:

Acquiring structured task representations derived from human demonstrations is critical for bimanual manipulation, particularly because the sequence of actions, the objects involved, and the geometry of interactions fluctuate widely between different executions. A primary difficulty involves simultaneously modeling the discrete semantic structure of tasks and the temporal dynamics of object-centric geometric relationships in a way that facilitates reasoning about task progression. To address this, we propose a graph-based task representation that integrates semantic-geometric data, jointly encoding object identities, semantic relationships between objects, and individual motion histories. This is achieved through a Message Passing Neural Network (MPNN) encoder paired with a Transformer-based decoder.

The encoder functions exclusively on the temporal scene graph to generate structured representations that are independent of specific action labels. Subsequently, the decoder utilizes action-context conditioning to predict future actions, the relevant objects, and their respective movements. This architectural decoupling results in task-agnostic representations, allowing the encoder to be reused across different robotic embodiments by simply fine-tuning the decoder on a limited robot dataset. Our evaluation across eleven bimanual tasks drawn from two distinct datasets reveals that the advantages of structured semantic-geometric representations over simpler sequence-based models become more pronounced as variability increases in action ordering and object involvement. At the deployment stage, a planner integrates action and motion predictions with learned Probabilistic Movement Primitives. This approach secured full task success in two real-robot bimanual experiments and surpassed the performance of graph ablations, standard Transformers, decoder-only models, and fine-tuned vision-language model baselines.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC