Title: KForge: Utilizing LLMs for Cross-Platform Kernel Generation in AI Accelerators

Abstract:

As production inference increasingly relies on a diverse array of accelerators, agentic pipelines have emerged that interleave reasoning, tool invocation, and multi-agent coordination. Because each of these stages exhibits unique compute and memory requirements, optimal efficiency dictates that they execute on the hardware best tailored to their specific profiles. This necessity introduces a significant systems engineering challenge: modern pipelines demand high-performance kernels across an expanding landscape of hardware backends and programming paradigms. Traditionally, hand-crafting these kernels has been a labor-intensive process requiring profound low-level expertise, a method that fails to scale as kernel complexity increases.

While recent efforts have employed Large Language Models (LLMs) for automatic kernel generation, the field continues to struggle with the intricacies of low-level code production and generalization across different backends. To address these limitations, we introduce KForge, a cross-platform framework centered on an iterative refinement loop facilitated by two cooperating LLM-based agents. The first, a generation agent, produces kernels and progressively enhances them using feedback from compilation processes and correctness checks. The second, a performance-analysis agent, interprets profiling data—sourced from both programmatic APIs and GUI-based tools—to issue recommendations that guide subsequent synthesis rounds.

This iterative process alternates between functional passes, which ensure candidate kernels achieve correctness, and optimization passes, designed to narrow the performance gap relative to hand-tuned baselines. We validated KForge on two distinct backends with varying levels of baseline reference availability. On the NVIDIA B200, the framework delivered a 2.12% improvement in end-to-end throughput over TensorRT-LLM when benchmarked on the gpt-oss-20b inference speed test. Meanwhile, on the Intel Arc B580, KForge generated Triton kernels that achieved a 5.13x geometric mean speedup compared to the faster of PyTorch eager execution or torch.compile. This performance gain was realized across 37 GEMM and tail-operations workloads from KernelBench Level 2, primarily through the implementation of operator fusion and mixed-precision execution.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC