Title: MyoSem: Bridging Electromyography and Natural-Language Action Semantics for Hand Action Understanding

Abstract:

Electromyography (EMG) serves as a critical sensing modality for wearable interaction, prosthetic control, and gesture recognition, as it directly captures muscle activation. However, conventional EMG approaches typically treat hand action understanding as a classification task based on fixed labels, which hinders the ability to perform querying, retrieval, and generalization using natural language action descriptions. To address this limitation, we introduce MyoSem, a framework designed to align EMG signals with action semantics by mapping low-level EMG data into a shared semantic space derived from multi-view action descriptions. MyoSem integrates three core components: the construction of multi-view action semantics, EMG encoding that accounts for activation patterns, and semantic query alignment. This architecture facilitates bidirectional retrieval between text descriptions and EMG signals.

We conducted a systematic evaluation of MyoSem using the EMG2Pose and NinaPro-series datasets. Our results demonstrate that MyoSem achieves strong performance in bidirectional EMG-text retrieval, consistently surpassing most baseline methods. Furthermore, the framework exhibits robust generalization capabilities across unseen users, held-out action classes, and transfer scenarios involving amputee users. Additional ablation studies and visualizations confirm the efficacy of each individual module. Ultimately, MyoSem shifts the paradigm of EMG-based hand action understanding from static, fixed-label recognition to dynamic, queryable bidirectional semantic retrieval, offering a novel approach to language-mediated EMG action analysis.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC