RT info:eu-repo/semantics/article
T1 EFAR-MMLA: An evaluation framework to assess and report generalizability of machine learning models in MMLA
A1 Chejara, Pankaj
A1 Prieto, Luis P.
A1 Ruiz Calleja, Adolfo
A1 Rodríguez Triana, María Jesús
A1 Kant Shankar, Shashi
A1 Kasepalu, Reet
K1 Multimodal learning
K1 Aprendizaje multimodal
K1 Machine learning
K1 Aprendizaje automático
AB Multimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA’s educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a “wicked problem” in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques.
PB MDPI
SN 1424-8220
YR 2021
FD 2021
LK https://uvadoc.uva.es/handle/10324/49171
UL https://uvadoc.uva.es/handle/10324/49171
LA eng
NO Sensors, 2021, vol. 21, n. 8, 2863
NO Producción Científica
DS UVaDOC
RD 22-dic-2024