Title: ScriptHOI: Learning Scripted State Transitions for Open-Vocabulary Human-Object Interaction Detection

Abstract:

Open-vocabulary human-object interaction (HOI) detection faces the challenge of identifying interaction phrases that were not included as annotated categories during the training phase. While recent vision-language HOI detectors have enhanced semantic transfer capabilities by aligning human-object features with text embeddings, their outputs are frequently skewed by object affordance and the co-occurrence of phrases. Consequently, a model might infer a cut cake interaction simply because a knife and cake are present, without confirming that the hand, tool, target, contact pattern, and object state collectively validate the action.

To address this, we introduce ScriptHOI, a structured framework that models each interaction phrase as a soft scripted state transition. Instead of processing a phrase as a singular class token, ScriptHOI breaks it down into distinct slots for body-role, contact, geometry, affordance, motion, and object-state. A visual state tokenizer converts each identified human-object pair into corresponding state tokens, while a slot-wise matcher evaluates both script coverage and script conflict. These metrics serve to calibrate HOI logits, highlight absent visual evidence, and establish training constraints for incomplete annotations.

Furthermore, to prevent the suppression of valid but unannotated interactions, we implement interval partial-label learning. This approach restricts unannotated candidates using script-derived lower and upper probability bounds, rather than labeling them as closed-world negatives. Additionally, a counterfactual script contrast loss is employed, which swaps individual script slots to mitigate shortcuts based solely on object presence. Evaluations on HICO-DET, V-COCO, and open-vocabulary HOI splits demonstrate that ScriptHOI enhances the recognition of rare and unseen interactions while significantly lowering false positives caused by affordance conflicts.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC