Title: Moving Past Scalar Rewards: The Power of Dense Feedback in LLM Policy Synthesis for Sequential Social Dilemmas

Abstract:

This research explores the synthesis of policies for language models (LLMs), a process that involves iteratively generating programmatic agent behaviors for multi-agent systems. Instead of relying on reinforcement learning to train neural policies, our approach utilizes LLM prompts to create Python-based policy functions. These policies are then tested in self-play scenarios and improved through iterative performance feedback.

A central focus of this study is feedback engineering—specifically, determining the most effective evaluation data to present to the LLM during refinement. We compare the efficacy of sparse feedback, which provides only scalar rewards, against dense feedback, which incorporates additional social metrics such as efficiency, equality, sustainability, and peace. Our experiments, conducted across two standard Sequential Social Dilemmas (Gathering and Cleanup) and utilizing two advanced LLMs (Claude Sonnet 4.6 and Gemini 3.1 Pro), demonstrate that dense feedback consistently performs as well as or better than sparse feedback across all measured criteria.

We attribute this performance advantage to a phenomenon termed "feedback aliasing." When relying solely on scalar rewards, distinct failure modes—such as under-cleaning versus over-cleaning—may result in identical reward values, confusing the model. Social metrics resolve this ambiguity by distinguishing between these scenarios, thereby enabling the LLM to identify the precise corrective action needed. Consequently, social metrics act as a vital coordination signal rather than a distraction, facilitating the development of sophisticated strategies like Voronoi territory partitioning and waste-adaptive cleaner schedules.

Code repository: https://github.com/vicgalle/llm-policies-social-dilemmas

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC