How to Evaluate Games in Education: A Literature Review

Overview

• Recent studies have examined the effectiveness of using game elements in higher education.
• Findings from these studies are varied, showing both positive and negative effects.
• This paper reviews and analyzes different studies to summarize and evaluate controlled experiments conducted in various scientific disciplines.
• The authors focus on the clarity of non-experimental conditions' descriptions in these studies.

Plain English Explanation

Many universities and colleges are incorporating game-like elements into their courses and programs. Researchers have been studying whether these "gamified" or game-based educational experiences are effective. The results of these studies have been mixed, with some showing benefits and others showing drawbacks.

In this paper, the authors review and analyze various research studies on this topic. They focus on how clearly the researchers describe the "control group" activities - the educational methods, materials, and starting conditions used for the non-game-based comparison groups.

The authors found that in most cases, the control group activities were not described well. It was often unclear what teaching methods and materials were used for those students. This makes it difficult to fully understand and compare the effectiveness of the game-based and non-game-based approaches.

The authors note that studies in computer science and engineering tend to report their results more clearly than studies in other fields. Based on these findings, the authors provide some recommendations for how future researchers can design their studies in a way that allows for more meaningful comparisons between game-based and traditional educational approaches.

Technical Explanation

The paper reviews empirical studies that examine the effectiveness of incorporating game elements into higher education. The findings from these studies have been mixed, with both positive and negative effects reported.

To better understand this body of research, the authors conducted a comparative meta-analysis. They reviewed controlled experiments across different scientific disciplines, focusing specifically on the clarity of the descriptions for the non-game-based "control group" activities.

The authors found that in most cases:

1. The educational methods used in the control group activities were poorly described.
2. The educational materials used in the control group activities were often unclear.
3. The starting conditions for the control and experimental groups were ambiguous.

This lack of clarity makes it difficult to fully compare the game-based and traditional educational approaches. Interestingly, the authors observed that studies in computer science and engineering generally reported their results more clearly than studies in other fields.

Based on these findings, the authors provide recommendations for how future researchers can design their empirical studies on game-based learning in a way that allows for more structured and meaningful comparisons.

Critical Analysis

The paper raises an important issue regarding the design and reporting of empirical studies on game-based learning in higher education. The authors rightly point out that the lack of clarity around control group activities makes it difficult to fully evaluate the effectiveness of incorporating game elements.

One potential limitation of the study is the scope - the authors only analyzed a subset of the available research, focusing on controlled experiments. There may be value in also examining less formal case studies or qualitative research that could provide additional insights.

Additionally, the authors acknowledge that their observations about differences in reporting clarity across disciplines warrant further investigation. It would be helpful to understand the underlying reasons for these disciplinary trends, which could inform best practices for all researchers in this area.

Overall, the authors raise an important methodological issue that deserves greater attention from the research community. By encouraging more rigorous and transparent reporting of study design and control group activities, future research on game-based learning can lead to more reliable and actionable insights.

Conclusion

This paper highlights the need for more clarity and consistency in how researchers design and report their empirical studies on incorporating game elements into higher education. The authors' analysis shows that in many cases, the control group activities are not described well, making it challenging to properly compare the effectiveness of game-based and traditional educational approaches.

The authors' recommendations for improving study design and reporting can help advance this field of research. By ensuring that control group activities are clearly documented, future studies will be better equipped to provide reliable, generalizable insights. This, in turn, can inform more effective implementation of game-based learning in colleges and universities.

Overall, this paper underscores the importance of methodological rigor in educational technology research. As the use of games and gamification continues to grow in academia, the research community must rise to the challenge of designing and reporting studies in a way that enables robust, meaningful comparisons.