Title: Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movements

Abstract:

To accurately model human behavior, it is essential to utilize a representation of bodily motion that leverages its inherent compositional nature. In this work, we introduce a hierarchical framework built upon "Action Atoms," which denote fundamental joint movements, and "Action Motifs," which arise from the temporal arrangement of these atoms to capture recurring body dynamics across diverse actions. We present A4Mer, a nested latent Transformer architecture designed to acquire this hierarchical structure directly from human pose data through a fully self-supervised approach.

A4Mer processes 3D pose sequences by dividing them into segments of varying lengths, encoding each segment as a singular latent token known as an Action Atom. Through a bottom-up learning process, temporal patterns constructed from these atoms—representing semantically meaningful and reusable spans of movement, or Action Motifs—emerge organically. This is accomplished via a unified pretext task involving masked token prediction within their respective latent spaces.

Furthermore, we present the Action Motif Dataset (AMD), a comprehensive collection of multi-view human behavior videos equipped with complete SMPL annotations. To address challenges posed by frequent and severe body occlusions during frame-wise annotation, we propose an innovative camera setup where devices are mounted on the subjects' feet. Our experimental findings confirm that A4Mer effectively extracts significant Action Motifs, thereby substantially enhancing performance in various human behavior modeling applications, such as action recognition, motion prediction, and motion interpolation.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC