Virtual Intelligent Agent for Hand Therapy Support
Abstract
This paper presents the design and implementation of an artificial intelligence driven virtual therapist that guides repetitive hand opening and closing exercises for rehabilitation—accessible to people of any age, from any location, on any webcam equipped computer. The agent integrated varies techniques for its quasi-natural interaction with the user, to determinate the correct advance during the therapy is supported on recurrent networks of long-short term memory (LSTM) is used. The network allows the processing of features obtained from hand movement samples, which contain information on positions, distances and angles, as well as time-controlled image captures for specific periods of time. These features are obtained using MediaPipe, an open-source framework developed by Google that allows discrimination of characteristic points of the hand. The proposed development focuses on performing a classification in time series samples of the opening and closing movement of the hand in real time, to define whether the user is performing correctly or not a basic exercise for strengthening hand joints. As contribution to agent’s finds, a large language model (LLM) is integrated for the analysis of the training results (hand exercise session) and the generation of a detailed report. The system is integrated into a web application developed using two frameworks that demarcate the creation of the agent. The design and user experience were developed using React, the management and data processing was performed using FastAPI. It is possible to evidence the easy use of the agent as a therapeutic assistant and an accurate progress report of the rehabilitation therapy, where according to an accuracy of 88% in learning of the LSTM, it accurately discriminates the movement in hand physical therapy.
Keywords: artificial intelligence, assistive robotics, physical therapy, long and short term memory (LSTM) networks, React agent, recurrent neural network.
Full Text:
PDFReferences
D. N. SOUMIS, and N. D. TSELIKAS. Serious Game for Hand Therapy Assessment, in 13th International Conference on Modern Circuits and Systems Technologies (MOCAST), Sofia, Bulgaria, 2024. https://doi.org/10.1109/MOCAST61810.2024.10615316
R. K. MEGALINGAM, S. K. MANOHARAN, S. M. MOHANDAS, C. P. K. REDDY, E. N. P. N. V. K. VIJAY, and Y D. CHANDRIKA. Wearable Hand Orthotic Device for Rehabilitation: Hand Therapy with Multi-Mode Control and Real-Time Feedback. Applied Sciences, 2023, 13(6): 3976. https://doi.org/10.3390/app13063976
G. DIRACO. Chapter 1 - Advanced sensors for smart healthcare: an introduction, in Advanced Sensors for Smart Healthcare, T. A. Nguyen, Ed., Elsevier, 2025, 1-27. https://doi.org/10.1016/B978-0-443-24790-3.00002-8
Z. LUO, J. PENG, X. ZHANG, H. JIANG, R. YIN, Y. TAN, and M. LV. Optimal scheduling of smart home energy systems: A user-friendly and adaptive home intelligent agent with self-learning capability. Advances in Applied Energy, 2024, 15: 100182. https://doi.org/10.1016/j.adapen.2024.100182
S. HOSSEINI, and H. SEILANI. The role of agentic AI in shaping a smart future: A systematic review. Array, 2025, 26: 100399. https://doi.org/10.1016/j.array.2025.100399
J. WANG. Multi agent system based smart grid anomaly detection using blockchain machine learning model in mobile edge computing network. Computers and Electrical Engineering, 2025, 121: 109825. https://doi.org/10.1016/j.compeleceng.2024.109825
M. HUBERT, S. KAZEMI, M. HUBERT, A. CARUGATI, and M. M. MARIANI. The exploration of users’ perceived value from personalization and virtual conversational agents to enable a smart home assemblage– A mixed method approach. International Journal of Information Management, 2025, 80: 102850. https://doi.org/10.1016/j.ijinfomgt.2024.102850
H. MANCY, N. E. GHANNAM, A. ABOZEID, and A. I. TALOBA. Decentralized multi-agent federated and reinforcement learning for smart water management and disaster response. Alexandria Engineering Journal, 2025, 126: 8-29. https://doi.org/10.1016/j.aej.2025.04.033
M.-H. YEN, N. MISHRA, W.J. LUO, and C.E. LIN. A Novel Proactive AI-Based Agents Framework for an IoE-Based Smart Things Monitoring System with Applications for Smart Vehicles. Computers, Materials and Continua, 2025, 82(2): 1839-1855. https://doi.org/10.32604/cmc.2025.060903
Y. WANG, S. SU, and Y. WANG. Attention-augmented multi-agent collaboration for Smart Industrial Internet of Things task offloading. Internet of Things, 2025, 31: 101572. https://doi.org/10.1016/j.iot.2025.101572
L. V. D. NASCIMENTO, and J. P. M. D. OLIVEIRA. A multi-agent architecture for context sources integration in smart cities. Future Generation Computer Systems, 2025, 172: 107862. https://doi.org/10.1016/j.future.2025.107862
G. SINGH. Wearable IoT (w-IoT) artificial intelligence (AI) solution for sustainable smart-healthcare. International Journal of Information Management Data Insights, 2025, 5(1): 100291. https://doi.org/10.1016/j.jjimei.2024.100291
B. RAVIKRISHNA, M. E. SENO, M. RAPARTHI, R. R. YELLU, S. ALSUBAI, A. K. DUTTA, A. AZIZ, D. ABDURAKHIMOVA, and J. BHOLA. Artificial intelligence probabilities scheme for disease prevention data set construction in intelligent smart healthcare scenario. SLAS Technology, 2024, 29(4): 100164. https://doi.org/10.1016/j.slast.2024.100164
S. M. SCHUELLER, C. STILES-SHIELDS, and L. YAROSH. Online Treatment and Virtual Therapists in Child and Adolescent Psychiatry. Child and Adolescent Psychiatric Clinics of North America, 2017, 26(1): 1-12. https://doi.org/10.1016/j.chc.2016.07.011
Y. P. P. CHEN, C. JOHNSON, P. LALBAKHSH, T. CAELLI, G. DENG, D. TAY, S. ERICKSON, P. BROADBRIDGE, A. E. REFAIE, W. DOUBE, and M. E. MORRIS. Systematic review of virtual speech therapists for speech disorders. Computer Speech & Language, 2016, 37: 98-128. https://doi.org/10.1016/j.csl.2015.08.005
A. MARKS, A. GARBATINI, V. P. KIMBERLY HIEFTJE, V. WESER, and C. S. F. FERNANDES. Use of Immersive Virtual Reality Spaces to Engage Adolescent and Young Adult Patients With Cancer in Therapist-Guided Support Groups: Protocol for a pre-post study. JMIR Research Protocols, 2023, 12: 48761. https://doi.org/10.2196/48761
J. MILLS, and O. DUFFY. Speech and Language Therapists’ Perspectives of Virtual Reality as a Clinical Tool for Autism: Cross-Sectional Survey. JMIR Rehabilitation and Assistive technologies, 2025, 12: 63235. https://doi.org/10.2196/63235
A. MILOFF, P. CARLBRING, W. HAMILTON, G. ANDERSSON, L. REUTERSKIÖLD, and P. LINDNER. Measuring Alliance Toward Embodied Virtual Therapists in the Era of Automated Treatments With the Virtual Therapist Alliance Scale (VTAS): Development and Psychometric Evaluation. Journal of Medical Internet Research, 2020, 22(3): 16660. https://doi.org/10.2196/16660
K. HAN, and S. KONG. Exploring factors affecting art therapists’ intention to embrace virtual reality using UTAUT1. The Arts in Psychotherapy, 2025, 92: 102260. https://doi.org/10.1016/j.aip.2025.102260
S. G. ROZEVINK, B. MAAS, A. MURGIA, R. M. BONGERS, and C. K. V. D. SLUIS. Therapists’ and prosthesis users’ assessments of a virtual reality environment designed for upper limb prosthesis control training. Journal of Hand Therapy, 2025. https://doi.org/10.1016/j.jht.2025.01.004
T. AHMED, E. L.-G. M. R. ISLAM, I. WANG, and M. RAHMAN. Robot-aided Rehabilitation with SREx: A Smart Robotic Exoskeleton for Reducing Therapists' Physical Stress. Archives of Physical Medicine and Rehabilitation, 2022, 103(12): 184-185. https://doi.org/10.1016/j.apmr.2022.08.936
T. H. NGUYEN, M. ELIOS, A. M. FITZPATRICK, S. M. NYENHUIS, and W. PHIPATANAKUL. Optimizing Pediatric Asthma Management: Implementation and Evaluation of AIR and SMART Therapies. The Journal of Allergy and Clinical Immunology: In Practice, 2025. https://doi.org/10.1016/j.jaip.2025.05.004
W. BAE, N. JUNG, Y. JUNG, and K. KAM. Analysis of the applicability of an artificial intelligence technology to classify hand movements of people with or without limited hand movement, using AI-motion SW: An observational study. Edelweiss Applied Science and Technology, 2025, 9(3): 2930–2940. https://doi.org/10.55214/25768484.v9i3.5889
T. V. P. CISNEROS, A. BRONS, B. KRÖSE, B. SCHOUTEN, and G. LUDDEN. Playfulness and New Technologies in Hand Therapy for Children With Cerebral Palsy: Scoping Review. JMIR Serious Games, 2023, 11(1): 44904. https://doi.org/10.2196/44904
G. BURDEA, N. KIM, K. POLISTICO, A. KADARU, N. GRAMPUROHIT, D. ROLL, and F. DAMIANI. Assistive game controller for artificial intelligence-enhanced telerehabilitation post-stroke. Assistive Technology, 2019, 33(3): 117–128. https://doi.org/10.1080/10400435.2019.1593260
D. N. SOUMIS, and N. D. TSELIKAS. A Web-Based Platform for Hand Rehabilitation Assessment. Big Data and Cognitive Computing, 2025, 9(3): 52. https://doi.org/10.3390/bdcc9030052
P. MANIAR, G. VEDPATHAK, and S. FRANCIS. VTherapy: Deep Learning-based Hand Rehabilitation System. In 2nd International Conference on Artificial Intelligence in Healthcare, Education and Industry (IDICAIEI), Wardha, India, 2024. https://doi.org/10.1109/IDICAIEI61867.2024.10842730
J. GONZALEZ, and W. YU. Non-linear system modeling using LSTM neural networks. IFAC-PapersOnLine, 2018, 51(13): 485-489. https://doi.org/10.1016/j.ifacol.2018.07.326
W. SIMOES, L. REIS, C. ARAUJO, and J. M. JR. Accuracy Assessment of 2D Pose Estimation with MediaPipe for Physiotherapy Exercises. Procedia Computer Science, 2024, 251: 446-453. https://doi.org/10.1016/j.procs.2024.11.132
A. LATRECHE, R. KELAIAIA, A. CHEMORI, and A. KERBOUA. Reliability and validity analysis of MediaPipe-based measurement system for some human rehabilitation motions. Measurement, 2023, 214: 112826. https://doi.org/10.1016/j.measurement.2023.112826
S. RANI, S. MASOOD, and D. R. RIZVI. A Bi-LSTM based approach for Compensation detection during robotic stroke rehabilitation therapy. Procedia Computer Science, 2025, 258: 3145-3154. https://doi.org/10.1016/j.procs.2025.04.572
A. WOLFF, G. WEINSTOCK-ZLOTNICK, and J. GORDON. SSc management – In person appointments and remote therapy (SMART): A framework for management of chronic hand conditions. Journal of Hand Therapy, 2014, 27(2): 143-151. https://doi.org/10.1016/j.jht.2013.11.004
Refbacks
- There are currently no refbacks.
