Title: Ego-METAS: An Egocentric Benchmark for Online, Multimodal, Energy-Efficient Temporal Action Segmentation

Abstract: For embodied agents to function effectively in the physical world, they require "always-on" environmental perception. This necessitates a selective approach to sensor usage, carefully balancing energy limitations against the need for task accuracy. While this capability is crucial for devices with limited resources, energy-aware perception has been largely overlooked, as previous research typically assumed access to unlimited computing power. To bridge this gap, we present Ego-METAS, the inaugural benchmark dedicated to Egocentric online Multimodal Energy-efficient Temporal Action Segmentation.

Ego-METAS establishes a comprehensive testbed comprising over 100 hours of untrimmed egocentric video sourced from EgoExo4D, CMU-MMAC, and CaptainCook4D. The dataset encompasses five distinct modalities: RGB, audio, gaze tracking, IMU data, and monochrome camera feeds. We define an online temporal action segmentation challenge in which models are required to dynamically choose which sensors to activate at every timestep, strictly within hardware-representative energy constraints.

Accompanying the benchmark, we provide unified data splits, refined annotations, pre-extracted features, and a varied collection of baseline routing policies. Our analysis reveals that optimal sensor routing is heavily dependent on specific scenarios. Furthermore, we find that current policy-learning methods, which are primarily optimized for trimmed video clips, fail to generalize effectively to continuous, untrimmed environments. Nevertheless, even basic dynamic fusion strategies, such as random routing, play a vital role in maintaining predictive accuracy while respecting tight energy budgets. Ultimately, Ego-METAS offers a standardized platform for advancing robust, cost-conscious policies essential for autonomous, always-on embodied AI systems.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC