Title: DeliChess: A Multi-party Dialogue Dataset for Deliberation in Chess Puzzle Solving

Abstract: Multi-party dialogue serves as a vital environment for investigating collaborative reasoning and decision-making; however, current datasets seldom address structured, complex reasoning tasks that require depth. To bridge this gap, we present DeliChess, a new dataset comprising group deliberation dialogues where participants work together to solve multiple-choice chess puzzles. In this framework, each member first attempts the puzzle independently, followed by a multi-party discussion, after which the group submits a revised collective answer. The dataset comprises 107 dialogues, complete with full transcripts, individual choices made before and after the discussion, and metadata detailing puzzle difficulty and move quality.

We assess performance through three metrics derived from chess engine evaluations, revealing that deliberation notably enhances group accuracy. Additionally, we examine the impact of probing utterances—messages designed to elicit proposals, justifications, or strategic reflection—using a classifier trained on previous deliberation data. Our analysis indicates that while probing increases the variability of group performance post-discussion, it does not consistently result in improved outcomes. Ultimately, our dataset provides a robust testbed for modeling group reasoning, dialogue dynamics, and the resolution of conflicting perspectives and opinions within a clearly defined strategic domain.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC