Title: FinTradeBench: A Financial Reasoning Benchmark for LLMs

Abstract:

Making financial decisions in the real world is a complex endeavor that demands the ability to reason across diverse data sources, such as company fundamentals extracted from regulatory documents and trading signals derived from price movements. While Large Language Models (LLMs) have recently gained traction among financial analysts for decision-making tasks, current evaluation benchmarks remain insufficient. Most existing financial question-answering datasets concentrate heavily on balance sheet metrics, largely neglecting the assessment of how stocks behave in the market or how trading signals interact with fundamental data.

To bridge this gap and harness the capabilities of both analytical approaches, we present FinTradeBench, a novel benchmark designed to test financial reasoning by synthesizing company fundamentals with trading signals. The dataset comprises 1,400 questions based on NASDAQ-100 companies, spanning a ten-year historical period. These questions are categorized into three distinct reasoning types: those focused on fundamentals, those centered on trading signals, and hybrid questions that require integrating multiple signal types.

To guarantee reliability at a large scale, we implemented a "calibration-then-scaling" framework. This approach integrates expert-crafted seed questions, response generation using multiple models, self-filtering mechanisms within models, numerical auditing, and alignment checks between human and LLM judges. Our evaluation of 14 different LLMs, conducted under both zero-shot prompting and retrieval-augmented generation settings, revealed significant performance disparities. While retrieval augmentation significantly enhanced reasoning capabilities regarding textual fundamentals, it offered minimal improvement for trading-signal reasoning. These results underscore persistent challenges LLMs face in handling numerical and time-series data, pointing toward critical areas for future advancement in financial intelligence research.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC