Title: ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models

Abstract:

Although Large Language Models (LLMs) have seen widespread integration into the healthcare sector, they continue to face substantial hurdles when navigating complex clinical decision-making processes. Current evaluation benchmarks predominantly focus on single-course scenarios, failing to provide a systematic assessment of LLM capabilities in multi-course contexts where a patient’s health status changes over time. To bridge this critical gap, we introduce ClinicalMC, a novel benchmark designed specifically for multi-course clinical decision-making.

The benchmark comprises 1,275 samples in Chinese and 5,804 in English, spanning four distinct phases of patient care: from initial admission through triage, first-course examination/diagnosis/treatment, subsequent multi-course assessments and treatments, to the final diagnosis. The dataset reflects varying levels of clinical complexity, with patients in the English subset undergoing an average of 5.11 clinical courses, compared to 3.42 for those in the Chinese subset.

To rigorously evaluate LLM performance, we developed a multi-agent evaluation framework incorporating agents representing patients, examiners, and doctors. Utilizing this framework and benchmark, we established two experimental configurations: a static, single-turn setting and a dynamic, multi-turn setting. We tested three distinct categories of models: closed-source LLMs (e.g., GPT5-mini), open-source LLMs (e.g., DeepSeek-V3.2), and specialized medical LLMs (e.g., HuatuoGPT-o1). Through comprehensive evaluation, our goal is to deepen the understanding of LLM capabilities within the medical field and facilitate their safe and effective implementation in healthcare environments.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC