Integrating behavior analysis with machine learning to predict online learning performance: A scientometric review and empirical study

Overview

• Researchers conducted a scientific analysis to review existing studies on using machine learning (ML) algorithms to predict online learning performance.
• The findings show that most studies apply ML methods without considering students' learning behavior patterns, which can compromise the accuracy and precision of the predictions.
• This study proposes an integration framework that combines learning behavior analysis with ML algorithms to enhance the accuracy of predicting students' online learning performance.

Plain English Explanation

The study aimed to improve the accuracy of predicting how well students will perform in online learning courses. Researchers first reviewed existing research in this area and found that most studies simply apply machine learning (ML) methods without considering the different patterns in how students actually learn. This can lead to less accurate predictions.

To address this, the researchers developed a new framework that combines analyzing students' learning behaviors with applying various ML algorithms. The idea is that by first identifying distinct learning patterns among students, the ML models can make more accurate predictions of performance within each pattern.

The framework was tested on a real dataset from the online learning platform edX. It was able to distinguish two main learning patterns: students with low autonomy and motivated students. The results show the framework achieved nearly perfect prediction accuracy for the autonomous students and satisfactory accuracy for the motivated students.

Additionally, the researchers compared the performance of this integrated framework to directly applying ML methods without the learning behavior analysis. The integrated framework, especially when using the XGBoost algorithm, consistently outperformed the direct ML approach. It also significantly improved accuracy for the motivated students and for the random forest ML method, which had been the worst performer.

The researchers also looked at which specific learning behaviors were most important in predicting performance within each student pattern using the LightGBM algorithm and SHAP values.

Technical Explanation

The researchers first conducted a scientometric analysis to systematically review existing research on using ML to predict online learning performance. They found that most studies apply ML methods without considering students' learning behavior patterns, which can compromise the accuracy and precision of the predictions.

To address this, the researchers propose an integration framework that blends learning behavior analysis with various ML algorithms. The framework first uses clustering analysis to identify distinct learning patterns among students. It then applies different ML algorithms, such as XGBoost and random forest, to predict performance within each pattern.

Applying the framework to an edX dataset, the researchers distinguish two main learning patterns: low autonomy students and motivated students. The results show the framework achieves near-perfect prediction accuracy for the autonomous students and satisfactory accuracy for the motivated students.

The researchers also compare the performance of this integrated framework to directly applying ML methods without the learning behavior analysis. The results consistently demonstrate the superiority of the integrated framework, particularly when using XGBoost. The framework also significantly improves prediction accuracy for the motivated students and for the worst-performing random forest method.

Finally, the researchers evaluate the importance of various learning behaviors within each pattern using the LightGBM algorithm and SHAP values.

Critical Analysis

The study provides a thoughtful approach to enhancing the accuracy of predicting online learning performance by incorporating students' learning behavior patterns. The integration framework appears to be a promising methodology, as demonstrated by the superior performance compared to directly applying ML methods.

However, the study is limited to a single dataset from edX, and further validation on other online learning platforms and contexts would be beneficial to assess the generalizability of the framework. Additionally, the researchers do not delve into the specific learning behavior features that were most influential within each pattern, which could provide valuable insights for educational practitioners.

While the results are encouraging, it would be important to consider potential biases or limitations in the data and clustering analysis that may have influenced the identified learning patterns. Exploring alternative pattern identification approaches or incorporating additional learner characteristics could also strengthen the framework.

Conclusion

This study presents an innovative integration framework that blends learning behavior analysis with machine learning algorithms to improve the accuracy of predicting students' online learning performance. By first identifying distinct learning patterns and then applying tailored ML models, the framework consistently outperformed direct ML approaches, particularly for specific student groups.

The findings highlight the importance of considering learners' behavioral characteristics when developing predictive models for online education. The framework's success suggests it could be a valuable tool for educators and platform providers to better understand and support students in virtual learning environments. Further research to validate the framework's generalizability and explore its practical implications would be valuable next steps.