Title: SHERLOCK: Enabling Dynamic Knowledge Adaptation for LLM-Driven E-Commerce Risk Mitigation

Abstract:

Navigating the highly adversarial landscape of e-commerce demands rigorous case investigations to uncover emerging fraud patterns. Traditionally, these manual inquiries involve laborious analysis of associations and couplings within multi-source heterogeneous data, a process that significantly bottlenecks operational efficiency. Although Large Language Models (LLMs) offer potential for automating such analyses, their practical application is often constrained by the intricacy of risk scenarios and the scarcity of long-tail domain expertise.

To overcome these obstacles, we introduce Sherlock, a novel framework that synergizes structured domain knowledge with LLM-based reasoning via three pivotal components. Initially, we build a domain Knowledge Base (KB) by distilling structured expertise from diverse knowledge sources. Secondly, we implement a specialized two-stage retrieval-augmented generation strategy for case investigations. This approach merges input contextual augmentation with a "Reflect & Refine" module, ensuring the KB is fully utilized to enhance analytical accuracy. Finally, we have developed an integrated platform for operations and annotation, fostering a self-evolving data flywheel.

By facilitating continuous system evolution through real-time hotfixes via KB updates and periodic logic alignment through post-training, the system effectively counters adversarial drifts. Online A/B testing at JD.com reveals that Sherlock achieves an 82% Expert Acceptance Rate (EAR) and boosts daily investigation throughput by 386.7%. Furthermore, a 90-day evaluation demonstrates that the data flywheel successfully mitigated performance decay resulting from shifting tactics on two separate occasions, lifting the EAR ceiling by approximately 3.5% through autonomous model updates.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC