Title: Decoding the Finetuning Prior: Recovering Exact Content Through Contrastive Decoding Diffing

Abstract:

While narrowly finetuned language models are known to memorize implanted information verbatim, the challenge of auditing a deployed model’s training content—without access to its weights or source data—remains unresolved. Although recent studies indicate that activation disparities between base and finetuned models contain readable signals regarding the finetuning domain, the current state-of-the-art method, Activation Difference Lens (ADL), yields only vague domain descriptions and demands complete "white-box" access to internal model mechanics.

In this work, we present Contrastive Decoding Diffing (CDD), a novel diffing technique that functions exclusively on output-level logit distributions. CDD requires no weight access, no specific layer selection, and no per-model tuning, yet it successfully recovers implanted facts. The method relies on three core strategies: circumventing chat templates to reveal the raw finetuning prior, initiating generation with maximally vague pre-fills, and amplifying the logit-space divergence between the finetuned and base models at every decoding step.

Using a single default configuration, CDD achieves verbatim recovery of implanted facts—including precise drug names, vote counts, physical measurements, and procedural details—across four distinct architectures ranging from 1B to 32B parameters. It consistently outperforms ADL, offering greater access efficiency while operating approximately 170 times faster.

Furthermore, CDD exposes unintended artifacts from data pipelines. For instance, a fictional persona introduced by an LLM data generator through mode collapse leaked into model weights; CDD successfully extracted this persona, marking, to our knowledge, the first demonstrated end-to-end fingerprinting chain linking a data generator artifact to model weights and finally to recovered output.

Validated in real-world finetuning scenarios, CDD achieved near-perfect recovery across all single-dataset non-CoT variants and accurately identified all four datasets in mixed-dataset settings. The ability of this grey-box method to surpass white-box baselines highlights its significant practical value for enhancing transparency and accountability within AI systems.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC