Title: Moving Beyond 'What' to 'How' and 'Why': Delivering LLM-Generated Retrospective Summaries of Passive Tracking Data to Remote Family Members

Abstract

The widespread adoption of ubiquitous computing has positioned multi-modal tracking systems as valuable tools for offering timely reassurance and awareness to remote family members (RFMs) of older adults, who are pivotal in care coordination. Despite this potential, synthesizing diverse data streams into coherent, high-level content—such as retrospective summaries—presents significant technical challenges. Although recent studies highlight the efficacy of large language models (LLMs) in interpreting multi-modal tracking data, there has been insufficient focus on crafting narrative accounts tailored to RFMs. These individuals hold deep personal knowledge and emotional stakes regarding the older adult’s well-being, yet they often lack visibility into daily routines and possess limited capacity for hands-on caregiving.

This study investigates the application of LLMs to produce retrospective summaries from multi-modal tracking data specifically for RFMs. We utilized and adapted the existing "Vital Insight" system to generate preliminary summaries across various dates and data availability conditions, employing these outputs as technology probes. Subsequently, we conducted interviews with 11 RFMs to collect their feedback. Informed by these insights, we re-engineered the system to implement a multi-layer, multi-agent, insight-driven summarization method. This new approach progresses from objective statistics and descriptions to enriched, context-aware narratives.

A follow-up survey with the same 11 RFMs compared the redesigned summaries against the initial versions, revealing significant enhancements in user satisfaction, perceived helpfulness, trust, and willingness to accept such summaries. The paper concludes with design implications for AI-generated summaries intended for RFMs and similar audiences, stressing the importance of facilitating a sensemaking transition for family members. This shift moves beyond merely reporting "what" data was collected, toward explaining "how" their loved one is faring and "why" certain patterns exist.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC