Title: Onboarding the Agent: Evaluating Learning, Exploration, and Task Management in Professional Contexts

Abstract:

While the rapid advancement of Multi-modal Large Language Models (MLLMs) has significantly propelled workflow automation, current research predominantly focuses on maximizing performance in static settings. This approach often neglects the robustness required for deployment in stochastic, real-world environments. We highlight three critical obstacles in this domain: managing dynamic task scheduling, conducting active exploration amidst uncertainty, and engaging in continuous learning from past experiences.

To address these gaps, we present \method{}, a dynamic evaluation framework designed to simulate a "trainee" agent navigating an unfamiliar environment through ongoing exploration. Departing from conventional benchmarks, \method{} assesses agent capabilities across three specific dimensions: (1) context-sensitive scheduling to handle streaming tasks with fluctuating priorities; (2) cautious information gathering through active exploration to mitigate hallucinations; and (3) continuous adaptation by extracting generalized strategies from dynamically generated, rule-based tasks.

Our experimental results reveal that state-of-the-art agents exhibit substantial weaknesses in dynamic settings, particularly regarding active exploration and continual learning. This study establishes a new framework for judging agent reliability, moving evaluation metrics away from static tests toward realistic, production-ready scenarios. The source code is accessible at https://github.com/KnowledgeXLab/EvoEnv

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC