Title: Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion

Abstract:

In hazardous settings like industrial plants and disaster sites, the intuitive and dependable remote control of Unmanned Aerial Vehicles (UAVs) and mobile robots is critical, as human operators often remain in unsafe conditions. Hands-free teleoperation addresses this challenge by boosting operator mobility and situational awareness, ultimately enhancing safety. Although vision-based gesture recognition has been investigated for hands-free control, its reliability frequently drops due to lighting changes, background clutter, and occlusions, restricting its practical use in real-world scenarios.

To address these vulnerabilities, we present a multimodal gesture recognition system that combines inertial measurement data—specifically accelerometer, gyroscope, and orientation readings—from Apple Watches worn on both wrists, with capacitive sensing signals captured by specialized gloves. Our approach employs a late fusion mechanism grounded in the log-likelihood ratio (LLR). This strategy not only boosts recognition accuracy but also offers interpretability by measuring the specific contribution of each data modality.

To facilitate this study, we developed a novel dataset featuring 20 unique gestures modeled after aircraft marshalling signals, which includes synchronized recordings of RGB video, IMU data, and capacitive sensor inputs. Our experiments show that the proposed framework matches the performance of current state-of-the-art vision-based baselines. However, it does so with substantially lower computational demands, a smaller model footprint, and reduced training times, rendering it ideal for real-time robot control. Consequently, we highlight the promise of sensor-based multimodal fusion as a robust and interpretable approach for gesture-driven teleoperation of drones and mobile robots.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC