Title: The Dialectics of Alignment: Leveraging Unsafe Knowledge for Dynamic Safety Routing

Abstract:

Current strategies for aligning large language models (LLMs) predominantly rely on erasure, a method that involves filtering out unsafe data or instructing models to strictly decline harmful requests. Although this technique successfully curtails immediate toxicity, it significantly narrows the model’s epistemological horizon. Consequently, this leads to overly cautious systems that issue unhelpful, generic refusals even when faced with sensitive but harmless inquiries. This study questions the prevailing dogma that unsafe data should simply be discarded. Instead, we advocate for a dialectical alignment framework, arguing that unsafe data contains valuable, domain-specific insights essential for generating nuanced, safe, and informative content.

To put this theory into practice, we present SafeMoE, a Mixture-of-Experts (MoE) architecture designed to segregate unsafe knowledge into domain-specific Low-Rank Adapters (LoRA experts). These experts are trained solely on harmful corpora. To transform these unsafe components into safe outputs, we employ a lightweight gating network trained on a small, carefully selected set of safe and informative responses. At inference time, this router dynamically manages the unsafe experts, guiding the generation process to utilize their deep domain expertise while strictly adhering to safety protocols.

Comprehensive empirical tests across rigorous safety benchmarks reveal that SafeMoE not only enhances safety—delivering a relative improvement of over 20% in safe response rates (an absolute gain exceeding 15%)—but also generates more informative replies, particularly when safety and harmfulness are critical factors. Additionally, the routing system demonstrates robust zero-shot generalization capabilities to new domains and broader safety tasks, without requiring domain-specific supervision. Our results indicate a significant paradigm shift in alignment methodologies: genuine safety is achieved not by concealing unsafe knowledge, but by integrating it in a controlled manner.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC