Title: CORE-MTL: Re-evaluating Gradient Balancing Through Causal Orthogonal Representations

Abstract:

Multi-task learning (MTL) seeks to build a unified model for various tasks by utilizing a shared representation across different domains. Current optimization-focused strategies typically address this by balancing task gradients or altering the shared architecture. Yet, because these techniques do not account for the specific content of the shared representation, they struggle to separate genuine task-relevant structures from spurious contextual noise. This oversight often results in negative transfer and suboptimal generalization.

To address these shortcomings, we introduce Causal Orthogonal Representations for Multi-Task Learning (CORE-MTL). This framework is driven by causal principles and focuses on the representation itself, promoting a structured factorization into semantic and residual components. Specifically, it confines task-relevant structures within the semantic stream while directing nuisance variations to the residual stream. In the visual domain, we implement this framework by applying physical priors to structured scenes and statistical constraints to attributes.

From a theoretical standpoint, CORE-MTL offers a tighter out-of-distribution generalization bound compared to optimization-centric methods. It also mitigates task gradient interference without the need for explicit gradient projection or reweighting. Empirical evaluations demonstrate that CORE-MTL consistently surpasses existing approaches on visual multi-task benchmarks, performing robustly in both in-distribution and out-of-distribution scenarios. The code is accessible at https://github.com/Hope-Rita/CORE-MTL.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC