Title: Back to the Feature: Explaining Video Classifiers with Video Counterfactual Explanations

Abstract:

Counterfactual explanations (CFEs) consist of minimal, semantically relevant alterations to a model’s input that result in a change to its predictions. By pinpointing the critical features driving these decisions, CFEs offer contrastive interpretations for classifiers. While state-of-the-art visual CFE techniques have largely concentrated on image classifiers, the field of video models remains significantly under-researched. For video CFEs to be practically useful, they must adhere to physical plausibility, maintain temporal coherence, and display smooth motion trajectories. Current image-based CFE methods are ill-equipped to produce such temporally consistent, smooth, and physically realistic video explanations.

To bridge this gap, we introduce Back To The Feature (BTTF), an optimization framework designed specifically for generating video CFEs. Our approach incorporates two key innovations: first, an optimization scheme that retrieves initial latent noise conditioned on the input video’s first frame; and second, a two-stage optimization strategy that facilitates the search for counterfactual videos in the immediate vicinity of the original input. Both optimization processes are driven exclusively by the target classifier, ensuring the faithfulness of the explanations. Furthermore, to speed up convergence, we employ a progressive optimization strategy that gradually increases the number of denoising steps.

Extensive evaluations on video datasets, including Shape-Moving (for motion classification), MEAD (for emotion classification), and NTU RGB+D (for action classification), demonstrate that BTTF successfully generates valid, realistic, and visually similar counterfactual videos. These outputs provide concrete insights into the underlying mechanisms of the classifier’s decision-making process.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC