Development of a Hand Gesture Detection System Based on MediaPipe for Neuromotor and Cognitive Rehabilitation Using Augmented Reality on the M3Display Platform

Abstract

This Final Project focuses on the design and development of a serious augmented reality-based game for rehabilitation of upper-limb mobility and fine motor functions. The game-based approach to rehabilitation aims to make the recovery process more engaging and enjoyable for the patient, and the usage of augmented reality aims to facilitate this effect by making the game more immersive. The game is developed using Unity game engine and C# programming language. As its base, the game uses M3Display - an augmented reality-based system for the rehabilitation of upper limbs functionality and mobility, together with the real-time hand segmentation solution previously implemented for this system and based on the MediaPipe neural network.