Title: Enhancing Deep Equilibrium Models Through Consistency

Abstract:

Deep Equilibrium Models (DEQs) have established themselves as a robust paradigm within deep learning, distinguished by their capacity to represent infinite-depth architectures while maintaining constant memory consumption. Despite these advantages, the iterative nature of fixed-point solvers imposes substantial inference latency on DEQs. To address this challenge, we propose the Consistency Deep Equilibrium Model (C-DEQ), a new framework that utilizes consistency distillation to speed up DEQ inference. We conceptualize the iterative inference process of DEQs as a progression along a fixed Ordinary Differential Equation (ODE) trajectory toward equilibrium. By training C-DEQs to reliably map intermediate states directly to the fixed point along this path, we enable efficient few-step inference without compromising the performance of the teacher DEQ. Additionally, this approach supports multi-step evaluation, allowing for a flexible balance between computational cost and performance. Comprehensive experiments across multiple domain tasks reveal that C-DEQs deliver consistent accuracy improvements of 2 to 20 times compared to implicit DEQs when subjected to the same few-step inference constraints. The source code is accessible at https://github.com/landrarwolf/CDEQ.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC