Multimodal Assistance in Rehabilitation: User Experience of Embodied and Non-Embodied Agents for Collecting Patient-Reported Outcome Measures

Download publication here.

The collection of patient-reported outcome measures (PROMs) is a key measurement tool for patient-centred care. At the same time, collecting these measures poses obstacles for many patients, leading to these groups being underrepresented in the data. We have therefore developed a multimodal, AI-driven assistance system to support patients in collecting these data. The interface of the system comprised a digital tablet containing the PROM questionnaire items and the assistant in three forms of embodiment: A virtual avatar, a physical avatar, and a voice-only agent. To evaluate the users’ experience and ratings of the system, two separate studies were implemented in two rehabilitation centers with 195 patients. A mixed within–between RCT was conducted at an outpatient clinic, where patients completed PROMs both with and without an assistant, and a between-subject design at an inpatient clinic comparing routine PC-based care with avatar- and robot-assisted PROM administration. Our results suggest a preference for the non-assisted tablet-only condition in Clinic A, whereas, in Clinic B, both agent conditions were preferred over routine care. We have further analyzed aspects such as trust and social presence in this study to gain a more thorough understanding of the users’ experience. Our analysis shows a higher trust rating for the voice-only assistant, whereas the robot, virtual avatar, and the voice-only conditions were perceived as more socially present. The impact of demographic factors and affinity for technology on the user ratings was also thoroughly studied. Our findings shed light on the role of agent embodiment in PROM assistance and contribute to the future design and evaluation of effective, engaging, and trustworthy systems for data collection in healthcare settings.

Recommended citation: Ashrafi, N., Graf, P., Marquardt, M., Harnisch, P., Hillmann, S., Ploner, N., Compagna, D., Cirit, E., Papst, L. & Voigt-Antons, J.-N. (2026). Multimodal Assistance in Rehabilitation: User Experience of Embodied and Non-Embodied Agents for Collecting Patient-Reported Outcome Measures. Virtual Worlds, 5(1), 15. https://doi.org/10.3390/virtualworlds5010015