RT info:eu-repo/semantics/article
T1 Lightweight real-time hand segmentation leveraging MediaPipe landmark detection
A1 Sánchez Brizuela, Guillermo
A1 Cisnal de la Rica, Ana
A1 Fuente López, Eusebio de la
A1 Fraile Marinero, Juan Carlos
A1 Pérez Turiel, Javier
K1 Augmented reality
K1 Hand segmentation
K1 MediaPipe
K1 Online processing
K1 Semantic segmentation
K1 33 Ciencias Tecnológicas
AB Real-time hand segmentation is a key process in applications that require human–computer interaction, such as gesture rec-ognition or augmented reality systems. However, the infinite shapes and orientations that hands can adopt, their variabilityin skin pigmentation and the self-occlusions that continuously appear in images make hand segmentation a truly complexproblem, especially with uncontrolled lighting conditions and backgrounds. The development of robust, real-time handsegmentation algorithms is essential to achieve immersive augmented reality and mixed reality experiences by correctlyinterpreting collisions and occlusions. In this paper, we present a simple but powerful algorithm based on the MediaPipeHands solution, a highly optimized neural network. The algorithm processes the landmarks provided by MediaPipe usingmorphological and logical operators to obtain the masks that allow dynamic updating of the skin color model. Differentexperiments were carried out comparing the influence of the color space on skin segmentation, with the CIELab color spacechosen as the best option. An average intersection over union of 0.869 was achieved on the demanding Ego2Hands datasetrunning at 90 frames per second on a conventional computer without any hardware acceleration. Finally, the proposed seg-mentation procedure was implemented in an augmented reality application to add hand occlusion for improved user immer-sion. An open-source implementation of the algorithm is publicly available at https://github.com/itap-robotica-medica/lightweight-hand-segmentation.
PB Springer
SN 1359-4338
YR 2023
FD 2023
LK https://uvadoc.uva.es/handle/10324/61780
UL https://uvadoc.uva.es/handle/10324/61780
LA eng
NO Virtual Reality, 2023.
NO Producción Científica
DS UVaDOC
RD 14-oct-2024