Title: FACT: A Simple and Efficient Framework for Active Finetuning

Abstract:

Active finetuning aims to enhance a pretrained model’s capability on a specific domain or task by adapting it to a curated set of informative or difficult samples. While prior studies have concentrated heavily on the active learning component—specifically data selection—they have typically relied on full finetuning for model adaptation. This standard approach often disrupts pretrained features caused by distribution shifts, a problem that worsens when the model parameters significantly outnumber the available finetuning data, thereby increasing the risk of overfitting.

To bridge this gap, we introduce the FiAF task, which calls for a systematic exploration of finetuning methods within active learning contexts. We present FACT, a streamlined and efficient three-phase hierarchical finetuning framework tailored specifically for active finetuning. Our extensive experimental evaluation covers four main areas:

1. Datasets: We tested on three categories of image classification tasks—classic (CIFAR10, CIFAR100, ImageNet-1k), imbalanced (CIFAR10-LT, CIFAR100-LT), and fine-grained (StanfordCars, FGVCAircraft)—using 3 to 5 different sampling ratios for each.
2. Architectures: We utilized diverse pretrained models, including ConvNeXt, Vision Transformer (ViT), and Vision LSTM (ViL).
3. Strategies: We conducted a systematic investigation into frozen feature augmentation (FroFA) techniques.
4. Analysis: We performed a rigorous assessment of the framework’s efficiency and generalizability.

The findings highlight substantial performance improvements characterized by strong robustness and generalization. Notably, at low sampling ratios, FACT delivered performance gains exceeding 20% on ViT models across the CIFAR10, CIFAR100, and ImageNet-1k benchmarks. This method establishes a new state-of-the-art while preserving parameter efficiency, proving especially valuable in scenarios where labeled data is limited.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC