Title: Typhoon: Developing an Effective Task-Specific Masking Strategy for Pre-trained Language Models

Abstract:

In the realm of masked language modeling (MLM), determining which tokens to mask remains a pivotal yet insufficiently explored design choice. While conventional pretraining approaches select tokens for masking via uniform random selection, emerging research indicates that employing more informative masking targets can enhance performance on downstream tasks. This study investigates masking as a task-adaptive element within the fine-tuning workflow and proposes Typhoon, a novel strategy that leverages the gradient of the task loss relative to one-hot token inputs to dynamically estimate the contribution of each token type to the objective function.

Typhoon operates by maintaining an exponential moving average of per-token-type saliency scores. These scores are then calibrated into a masking distribution, ensuring that the expected masking rate aligns with a predefined target budget, based on a token-independence approximation. To rigorously assess the method, we formalize the approach and compare it against standard random masking and whole-word masking. The evaluation spans two GLUE tasks (MRPC and CoLA), utilizing three BERT-family backbones (TinyBERT, DistilBERT, and BERT-base) and five random seeds for each configuration, resulting in a total of 90 training runs.

Our primary conclusion is that, after accounting for variance across seeds, no masking strategy demonstrates consistent superiority over the others for the evaluated tasks. Specifically, on the MRPC dataset, the performance difference between Typhoon and the strongest baseline remains within a margin of $0.004$ $F_1$. Furthermore, none of the twelve paired statistical tests involving Typhoon achieved significance, and every $95\%$ confidence interval included zero. Consequently, the preliminary advantages observed in single-run experiments do not withstand this more rigorous statistical evaluation. We interpret these findings as a cautionary, reproducibility-centered insight: gradient-based task-adaptive masking is competitive, yet it does not clearly outperform resource-free random masking at this scale. To facilitate future research, we provide a clean, modern reimplementation of the method.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC