Evaluation of awarding badges on Student’s engagement in Gamified e-learning systems

Abstract

Gamification has been gaining increasing acceptability in recent times in educational and commercially related activities, as a tool that encourages and improves the motivation of digital native learners. Since learners can easily engage, educationists have explored gamification as a tool for remediation of engagement, motivation, and collaboration. However, the literature showed that the structural and contextual deployment of game elements is defined only partially in practice. Subsequently, gamification success and failure factors should be explored to identify the required enhancement to achieve improved efficiency in current systems. This research extracts the relevant aspects of gamification that need due consideration to make a guided choice through existing theories. This study is based on an online gamified study that uses well-founded concepts in teaching and evaluation of students in a university. Although badges earned and time spent indicated an increase in engagement, the results show that further work needs to be done by incorporating feedback elements, social interaction, and interactive guidance. The underlying impression is that timely, frequent feedback and personalized guidance, avenues for collaboration and interactivity need to be explored towards the better utility of gamification. Therefore, learning culture in the current learner-centered environment should be further studied to infuse better productivity.

Introduction

The concept of using games for education (or gamification) has attracted many scholars as it addresses the growing concerns about engagement and motivation in the present technologically advanced times.. Gamification has been applied in several domains, including business and education, with several techniques (Ibáñez et al. 2014). The potential of Gamification as a promising intervention to improve motivation and engagement was first realised in the commercial activities, where organizational leaders realized that the lack of engagement on the part of employees could critically undermine smooth execution and positive outcomes sought of an innovative, and critical strategy (cognizant 20–20 insights 2013). Many studies have realized the potential of gamification in education to induce holistic and sustainable behavioral, affective, cognitive and social interactivity changes in the learning activity as a response to the changing learner needs (Deterding et al. 2011a; Gee 2003; Prensky 2001; Blohm and Leimeister 2013). It was averred that the mandatory requirement for sustainability in business success is continued engagement and motivation. Therefore, the informed gamification-based frameworks through means such as incentives and rewards for performance and innovation can help induce the behavioral and cognitive acumen needed for continued motivation and engagement.

Due to the emerging technology in our present times, the learner becomes a digital native (Prensky 2011) who processes information differently to that offered in the traditional learning environments. Studies showed that in the United States, about 58% of the population played video games (Eden 2016). Therefore, the solution is building an active learning construct that can be fostered successfully through gamification (Giannetto et al. 2013). Consequently, gamification could engage learners towards human growth (Gilbert et al. 2018); however, the evaluation of gamification in learning environment remains scarce in the literature.

Gamification has been applied using several techniques, which include points-scoring, leader boards, and awards of badges as rewards for completing levels of learning tasks (Poondej and Lerdpornkulrat 2016). In the educational context, rewards in the form of badges, points, and scores may act as external motivators to learners in gamified courses (Goehle 2013). Therefore, educators aim to make games a productive, motivating, and encouraging learners to strive for success (de-Marcos et al. 2014; O’Donovan et al. 2013). Most of the online education uses gamification in higher education; however, only a few studies are dedicated to training and tutorials (Hsin et al. 2013) to help students who need extra support in understanding and applying concepts learnt in regular classes at lower levels (K12) of academics. Realising the potential of gamification in improving engagement and motivation of learners (Karagiorgas and Niemann 2017; Fogg 2009; Nakada 2017), the objective of this study is to observe the effect of gamification on the engagement of learners in learning pursuits offered through an online learning environment to help students prepare for admission tests they take for higher education.

The preparatory course study spans 45 h that is composed of gamified quiz sections and non-gamified learning material. The non-gamified and the gamified parts are separated; 60% of the course duration is devoted to the non-gamified while 40% for dedicated to the gamified part. The present course aims to prepare the student for entrance exams to institutions of higher education. The course was implemented as an online training program based on the observed theories in the literature. A theoretical gap was identified concerning the implementation of a gamified approach to learning and development, where only about 10% of training and tutoring courses offered implemented gamification (Hordemann and Chao 2012).

The objective of this work is to evaluate the widespread and common rewards of badges for accomplishing learning tasks in a structured training schedule. The composition and design of the gamified version depend on the five-step educational model applied in education adapted from (Hsin et al. 2013). The guiding research question of the work in this paper is, “Does gamified activity using badges encourage students to be engaged in the online learning process?”. Therefore, this study uncovers the strength, deliberation required, and the limiting and possibly overlooked factors while deploying gamification using badges. Since, the role of gamification, as found in the literature, is not limited to the motivational and engagement factors, the proposed model seeks to explore the strength embedded in gamification towards summative as well as a formative assessment.

The paper is structured as follows. First, an overview of gamification is provided in section “Literature review”. In section “Methodology”, we described the design of the study. The next section presents our results and reports the findings. Finally, we conclude with a discussion of the implication of our design and suggestions for future work.

Literature review

The concept of Gamification is still in its infancy as a tool for improving performance and outcomes, which trigger the fun element to assess, analyze, and direct human behavior. There has been a debate about the exact definition of gamification. Defining gamification is compounded by the differentiated perspectives to trends, practices, learning situations, and as a process of information treatment (Koivisto and Hamari 2014). Therefore, gamification embedded in real-time feedback possesses the wherewithal to address diverse activities in education and entertainment concurrently, while also driving engagement through collaboration, interaction, competition, and social connectedness (Menezes and De Bortolli 2016). This study adopts the definition of gamification, which states that “gamification is the application of game mechanics in non-game related contexts.” (Deterding et al. 2011b).

Gamification and motivation

Although gamification is a mode of entrainment for learners, the objective is to be used as a guide for understanding the proficiency, and motivational constructs in different educational contexts (Sailer et al. 2014). The perspectives and implications for the twenty-first-century learner can help the game designer assess the game elements and their strengths that need to be deployed to maximize gains. Table 1 describes five essential perspectives of gamification motivations. The perspective indicates learner’s motivational constructs and implications indicates the specific tool or framework to address each ‘Perspective’.

Table 1 Motivation: Perspectives and implications (adapted from (Flores 2015))

Though widely perceived as a limited way to engage students and improve motivation towards better learning impacts, gamification corresponds to following dimensions (Hsin et al. 2013):

• a) Mastering Skills,

• b) Challenging environment,

• c) Create guidelines on the progress of educational pursuit,

• d) Student engagement through the arousal of interest,

• e) Maximize the outcomes of learning through the immersive process,

• f) Evokes positive changes in behavior towards learning and academics, and.

• g) Social collaboration and interaction.

The study in this paper takes cognisance of the factors and hence divides the learning into gamified and non-gamified parts to evaluate the effect of awards (badges) on the motivation and engagement to do better. According to the scholars (Hsin et al. 2013), none of the papers defined the pre-requisite skill conforming to disabled, high school or preschool levels. This observation has been pointed out in the present paper while designing the scope of the study of gamified content and measure the impact of awards (badges) on the learning accomplishment.

Proposed Gamification approach in education

This research adopted a structured sequence of five steps when applying gamification to educational constructs, as depicted in Fig. 1. Following these steps proves to be an efficient way to deploy game-elements while achieving success in meeting the objectives of improving engagement and motivation towards better learning practices. Apart from engagement and motivation, there is a need to get instant knowledge sharing, interactivity. Moreover, feedback during the execution of the process itself is essential rather than waiting for the end of the unit, semester, or term, which is a case of the traditional approach.

Fig. 1

Modeling gamification to education (adapted from (Hsin et al. 2013))

Therefore, the five-step model involves accumulating all possible relevant knowledge regarding the audience contextually to the instruction being imparted in the proposed gamified course. It is crucial to know the levels of knowledge and skills, the duration of the course, the learning environment, and the group size. Some of the critical data revealed by extracting such information pertain to the psychological factors at play, the learning environment, skills, motivational level, and focus amongst a targeted group of learners. Such information helps the designers and educators to choose the right mix of game-elements to obtain best results (Hordemann and Chao 2012).

The learning objectives comprise of three goals: instructional, learning, and behavioral goals. The gaming elements chosen should have the right combination and proportion of each of the objectives to be met to achieve the desired objectives. The third stage of structuring the experience implies the feedback element that needs to address the frequency, scaling, and motivational factors. Proper implementation of this step ensures that the learner is guided sequentially to higher learning levels through an immersive set of achievable goals.

The first four stepsenumerated above are imperative to decide whether, with the resources available, the designer of contextual educational content can assimilate valuable feedback, promulgate rules, assign differentiated levels, and assign tracking mechanisms. Therefore, when applying the last step to complete the gamification of educational content, the designer takes into consideration two significant achievement aspects benchmarks- self and society. The self-category is the encouragement required by the assessment process to improve upon his existing skills by earning badges, points, and meeting activity schedule norms. The social aspect of achievement is the public display of personal successes that posit the learner against other learners ensued by interaction and collaboration (Hordemann and Chao 2012).

Methodology

In this research, the student accomplishment parameters that incentivize learning and success instantly is a vital virtue embedded in the Moodle learning management system, which is considered free open-source and widely used online educational system. In the online environment where players engage with passionately, badges are amongst the most coveted awards that are shown publicly. Such awards, in the form of customized badges, also help measure the progress made by students as they acquire higher skill levels.

However, award systems suffer from two main issues (Hordemann and Chao 2012). First, the awards earned the least useful ranking amongst students voting for different game-elements. Second, it challenging to estimate the accurate timing and parameter mix to offer awards to learners. Therefore, all possible factors of measurement quality, quantification, and frequency, mandatory are critical to success (de Sousa Borges et al. 2014).

In this paper, accounting for the previous issues (Hordemann and Chao 2012), the learning part does not carry any awards, whereas only the quiz part that matches the actual exam in the entrance tests of various institutions has been gamified with awards in the form of badges.

Figure 2 shows the proposed methodology of this research. First, the gamification design is completed following the five-step model presented in section “Proposed Gamification approach in education”. Then the researcher applies the participant inclusion criteria to filter which participants should the study include. After that, the data collection commences the required data and statistics of students followed by correlation calculation to deduce results as part of the gamification evaluation process.

Fig. 2

Research methodology

Gamification design

The present study implemented a single case study, which was a course offered through the Moodle learning management system. The course structure of the course encompasses six main components. The first one is about the general requirements and rules of the course and the collection of the learner’s information. The second part of the course provides the learners with information concerning the educational institutions and the selection processes by providing them with readings and hyperlinks and videos. The remaining four stages addressed the following subject areas: mathematics, science, humanities, and languages. Within each subject, five quizzes were administered as tests.

Therefore, this study has six stages that correspond to badges 4–9.Upon the accomplishment of each stage, learners earned different badges, the first four badges are related to the completion of the learner profile, the conscious player, knowledge about the university, the fifth badge is the mathematics expert, the sixth is scholar of nature, seventh is learner of humanities, and the eighth is scholar of languages, and last is ready for quizzes badges.

The training section of the study is separated into two parts, which are the non-gamified and the gamified one. The first part of the non-gamified learning consisted of learning material which the student should grasp before the gamified part of such as lecture notes, videos, and instructional content. Students are expected to acquire an understanding of the instructional material provided throughout the non-gamified part in order to be able to play the gamified part of the course. Therefore, the study lines up with literature (de Sousa Borges et al. 2014) that there are difficulties in measuring formats of instructional content. Henceforth, the present study will aim at investigating the impact of gamification on learning and development from the standpoint of student engagement.

Participants’ inclusion criteria

Online subscriptions for this course were filled by 1423 learners who exhibited their interest in the offered learning experience. The participants were free to complete their online courses according to their convenience and pace. The data of the participants who did not accomplish the courses were eliminated as a way to cope with reliability issues. The remaining population was subjected to inclusion in order to allocate learners for detailed analysis. The inclusion criteria which were implemented are as follow:

• 1- Only learners with a minimal engagement time of 240 min were chosen.

• 2- Students who have attempted all six stages of the gamification.

• 3- Students who passed at least one of the quizzes.

The first criterion guarantees that all learners have been engaged thoroughly in the gamification have provided all the necessary information, were familiarized with the structure, and most prominently for the condition to fit the model incorporated in the analysis section of this research.

The second criterion was set in order to ensure that this research can measure their engagement throughout the learning process and to get a holistic view of their motivational scale. Moreover, it was incorporated to make sure that all students have earned badges as they progress throughout the gamification stages.

The third criterion was established to ensure that the participants made resilient efforts to advance through the course. Henceforth, engagement, and badges earned would be more reliable quantifiers for this research. After the inclusion criteria were set, only 369 adhered to the established criteria.

Data analysis

Following the literature, the target population of the present study was screened adequately to be experimented to draw a more precise understanding of the current levels of engagement aspects (Welch et al. 1992). This study uses badges as awards as incentives for successful completion of quizzes and other parts answered by the students (Hickey and Schenke 2019).

The students are awarded initially for the non-gamified section that comprises mainly of personal background information, understanding various aspects of the institution’s students seek admission to, the structure of the offered training course, and for interacting with the course coordinator and peers. The initial interaction, when completed, can earn the students three badges. Since this activity is not an accurate reflection of the engagement in the actual learning process, though it measures the motivation of the students, a bonus of the fourth badge is offered for those that do it by at least 10% of the allotted 240 min of engagement in preparing for the real learning process. Earning the four badges unlocks the content and the quiz sections of the training course to the aspiring students.

In this research, the quiz section is the only gamified part. Taking tests in the quiz section can earn badges to respective students. Each of the four subjects has five quiz sections preceded by relevant, preparatory content. Completing at least one quiz section in the training course subjects carried an award. Successful completion of all quiz sections of all subject matters carried an additional, bonus award. Thus, the total number of awards on offer is five awards for the course content and four awards for the background information and assessment of motivational aspects.

The aim here was to explore the relationship between student engagement measured and the total number of badges earned through the implementation of a statistical model based on the calculation of Pearson’s r between the variables of engagement and badges earned; then engagement and time spent on each stage. The aim here was to explore the existence of a significant relationship between engagement and time and badges earned. The badges have been chosen as awards as incentives for successful completion of quizzes and other parts answered by the students.

Result and discussion

The gamified and non-gamified parts of this study were measured to enable comparison; understandably, each part contributed equally significantly to the learning process using badges as rewards. Table 2 shows the distribution of badges included students.

Table 2 Distribution of badges of included students

The null hypothesis for the research question: gamification does not increase engagement of students on online courses.

The overall average of engagement time across subjects and awards earned was more in gamified to non-gamified mode. Students get little motivation compared to entertainment imposed by gamification. Therefore, the student spends more time studying and gaming more badges, which ultimately increase their performance.

For each group of students that satisfy a certain number of badges, this research measures the Pearson correlation between engagement and badges. Pearson’s r intends to measure the strength of the relation between two variables. The value of r lies in the range between − 1 and 1, the closer r to 1, the stronger the correlation and the closer to 0 the weaker.

The first part of the research consisted of testing the relationship between the engagements, which was measured throughout the six stages of the gamified section, the number of badges earned by each student. In order to explore the relationship, Pearson’s r correlation model was implemented. Figure 3 highlights the results generated after testing the variables of engagement on each stage and the number of badges earned throughout each stage of the gamified part of the online learning course.

Fig. 3

Gamified versus non-gamified scenario

Figure 3 stated that for the first stage, the engagement of the students and the number of badges earner exhibited a positive relationship with r = 0.50 at a significant level of (alpha< .05). As students were in the initial processes of providing personal information, levels of engagements were at their lowest; the gamification activity had not started yet. Thus, the relationship was observed to be moderate. As students started to get a sense of the gamification part, the relationship between engagement and total badges reached its peak with a strong and significant positive relationship with an r = 0.83, at stage 5. Thus, learner’s curiosity was triggered; both engagement and badges earned were higher.

Results suggest that the learner’s engagement was increasing by the badges earned; therefore, gamification processes can be inferred as a strategy to encourage learners and harvest their motivation. In stage 5, a relatively lower correlation can be inferred because most students do not like math, which was in that phase. Moreover, it implies that educators could work more at this stage to enhance the gamification elements. In the last stage, a relatively lower correlation implies that few students reach that stage, and in fact, most of them were repeating the quizzes to pass the final stage in an uncontrolled manner.

Henceforth, the exploration of the relationship between badges earned and engagement suggests that the total of the badges earned influences and impacts the engagement of the students throughout the whole stages of the games. Gamification encourages students to be fully engaged in the online learning experience.

Figure 4 highlight a holistic view of the average badges earned through each stage and the average engagement score. The first stage noted the earning of four badges with the lowest engagement score noted. Time is considered as a measure of engagement while learning. Students earned four badges, and their engagement was at their lowest. Results recommend that before the gamification process, students were less engaged and motivated about the learning experience. The engagement scores and the badges earned started to increase throughout the second, third, fourth, and fifth stage where it reaches the peak. The results propose that students were getting more involved in the online learning processes and the gamification as demonstrated by the average score of engagement and the average number of badges earned.

Fig. 4

Correlation of Student engagement and earned Badges

Therefore, the results prove that gamification does boost and engage students in the online learning experience. The final stage is when the students earned more badges; however, the engagement scores decreased slightly, which means that the students struggle hard in the last phase to get the final badge. Still, it can be noted that the exploration of the relationship between the number of badges earned and the engagement score can be sufficient to prove that learners are more engaged by the gamification approach through the online learning experience.

It is essential to get the final correlation figure for the whole experiment. Since each stage has a different distribution, we have adopted the Fisher z-scores transformation (Fisher 1921) to normalize the Pearson’s r scores. After scores were converted to z-scores, they were averaged and converted back to a single Pearson correlation. The final reported figure was (r = 0.69, alpha < 0.05). Consequently, a strong positive relationship exists between awards in badges and student engagement in gamification mode.

Moreover, sustained engagement is indicated by persistence and intense efforts in the entire learning process (Skinner and Belmont 1993). The current online learning demands self-regulation, monitoring, assessment, and complete autonomy on the part of the learner. Such disciplined behavioral changes should be inculcated in course design by teachers that offer online teaching/training (Barak et al. 2016) in this paper, that element was missing and should have been incorporated through thoughtful design.

In the non-gamified, this study observed two critical issues. First, data generated by feedback and faster response and intermittent interactions can help students in finding the practical and cognitive utility of the material placed before gamified activities, thereby help them organize their thoughts and use their time productively (Mandernach and Garrett 2014; Guo et al. 2014). Second, interactive feedback is the normative expectation of online learners, as they are deprived of face-to-face interaction in their learning process that can lead to disengagement easily (Rutz and Ehrlich 2016). The critical issues regarding feedback are nature, targeting, and timing of feedback (Mandernach and Garrett 2014). The observations imply the significance of formative assessment in online courses and their success in provoking student engagement and motivation. The inference is then that Summative evaluations do not suffice as a measurement criterion for engagement in online learning. It is not difficult to achieve timely, interactive, cognitive feedback in online teaching, as the presence and reach of the internet is supportive of such actions (Mandernach and Garrett 2014). Therefore, teachers assistance in presenting educative and interactive feedback motivates them to engage with the online course (Flores 2015).

Limitations

The study has been designed for a small cohort that is preparing for admission tests to colleges. Most of the study material presented here has already been taught in regular classes. This course aims to help students who need further help and practice to gain confidence to take the exams successfully.

The course has been designed to encourage students to indulge in self-study. No tutor support is provded, neither have the students been provided with collaborative tools such as messaging. This study was limited in regard to the social collaboration engagement parameter (Flores 2015). Incorporating the leaderboard could have also been considered. However, it was thought that the short duration of 45 h needed concentrated effort, and that including leaderboard and discussion forum platforms would distract the learners.

Conclusion

This paper has sought to uncover the possible oversights and corresponding gamification towards engagement in gamified online courses. The study relied on a quantitative approach to measure the engagement of learners throughout the gamification process through a case applied on an online course on Moodle. Results suggest time and badges earned to be a reliable measure of engagement in the online learning experience. This study found that students were more engaged as they advance throughout the course; therefore, the gamification process encouraged students to be more involved in the online learning experience. The main takeaways from the experiment appear in the form of missing ingredients, such as feedback issues, social interaction, and interactive guidance, which may affect engagement significantly. This paper found that the summative assessment may not suffice as a measurement criterion for sustained engagement. Therefore, formative assessment and interactive feedback should be incorporated throughout the online teaching and learning process.