Title: HERO'S JOURNEY: Evaluating Complex Rule Induction via Text-Based Games

Abstract: This paper presents HERO'S JOURNEY, a novel benchmark designed to test rule induction within goal-oriented episodic environments. In this framework, agents are required to deduce implicit rules from provided demonstrations and subsequently apply them through multi-step actions. The benchmark comprises eight distinct tasks spanning both attribute and procedural induction categories, featuring four specific structural rule forms, adjustable lexical grounding, and defined identifiability conditions. Our evaluation of current state-of-the-art Large Language Models reveals that while these models demonstrate some capacity for rule induction, their performance is constrained and inconsistent across different tasks. We observe that the process of execution creates a significant bottleneck for models, whereas surface-level semantics have little impact. Although steering techniques tailored for induction enhance performance on attribute-based tasks, they fail to yield consistent improvements on procedural tasks, indicating that the disparity in procedural induction capabilities remains an unresolved issue.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC