Incorporating personalized learning in a role-playing game environment via SID model: a pilot study of impact on learning performance and cognitive load

While role-playing games and personalized learning have been regarded as effective tools to improve students’ learning, incorporating personalized learning into role-playing games is challenging and approaches are limited to cognitive and motivational variables. Aiming at expanding approaches to incorporate personalization into role-playing games, this study included affective and cognitive variables to develop a personalized role-playing game, guiding by the situational design model. A pilot study was conducted to examine the effectiveness of the game on students’ learning performance and cognitive load. Results showed that personalized role-playing game environment was effective in improving students’ performance, reducing extraneous load, and promoting germane load. This study also found that although decreased extraneous load, could leave students more GL capacity for efficient learning, this would not necessarily lead to performance improvement. Students need to be motivated to invest sufficient germane load to actively process the learning materials and thus, improve performance. The findings have several implications for future research designing personalized educational games aimed to promote efficient learning.


Introduction
As technological advances in educational games, there is an increasing research interest in exploiting digital role-playing games (RPGs) in educational practice as RPGs are more likely to support a wide range of educational potential than many other games by allowing students to experience a series of educational scenarios and operate an avatar character in an imaginary environment (Daniau, 2016;Deterding & Zagal, 2018). Being designed for interaction, educational RPGs have given students the choice of determining their actions and provided an engaging approach for students to test their knowledge and reflect on the effect their choices made on the game (Rahman & Angraeni, 2020).
Personalized learning (PL) has been regarded as an effective pedagogical strategy to improve scholastic path, learning process, and learner satisfaction in different learning environments by providing a unique learning experience through accommodating individual differences in learning (Martin et al., 2020;Zhong & Xu, 2019;Zhong, 2022a). As perceiving the importance of addressing individual learners' needs in gaming environment, game designers began to incorporate PL into games by adaptively tailoring game content (Zualkernan et al., 2010), game structure (Lin et al., 2013), and/or presentation of game materials to individual game player (Soflano et al., 2015). Previous studies have proved the effectiveness of personalized games in improving learning . For example, Plass et al. (2019) compared the effectiveness of an adaptive and a non-adaptive version game on training shifting skills and results showed that the adaptive version was more effective than the non-adaptive version game in promoting student's performance. Clark et al. (2016) reported similar results that students who received adaptive self-explanation prompts significantly outperformed in the post-test than students who were in the non-adaptive version.
However, approaches to incorporate PL into educational games limited to cognitive and motivation variables, such as prior knowledge and motivation (Plass & Pawar, 2020). Those limited variables resulted in narrow approaches to personalize learning in educational games. More learner attributes, especially from affective and sociocultural domains, should be taken into consideration in the design of personalization in educational games. Additionally, PL has been rarely applied in the context of RPGs. Discussions of incorporating personalized learning specific in RPGs are quite limited in extant literature. Despite the pedagogical benefits of personalized RPGs, how PL could be best incorporated into RPGs environment is still unclear. For instance, will the incorporation of PL into RPGs impact students' performance? Will it require extra cognitive load from students? Little information can be drawn from extant literature to answer those questions. There is a need for more improvement to this research direction.
This study aims to fill this gap by exploring how PL could be incorporated into RPGs and whether the incorporation would affect students' learning, such as performance and cognitive load. The purpose of this study was to expand the approaches to incorporate PL into educational games by including affective variables in the design of personalized games. This study would provide empirical evidence of the effectiveness of personalized educational games on students' performance and cognitive load in the context of role-playing game. Findings of this study would assist researchers, especially international scholars who are interested in personalized game design as well as inclusive game design. The following questions guided this study: 1. How does a personalized RPG environment developed via situational instructional design model affect student's learning performance? 2. How does a personalized RPG environment developed via situational instructional design model affect student's cognitive load? 3. Is cognitive load related to learning performance in a personalized RPG environment?
The structure of this article is as follows. First, related concepts and frameworks were reviewed in the Literature section, including cognitive load, digital RPGs, PL and RPGs, and situational instructional design (SID) model. Second, the six steps of personalized RPG development were detailed in the Development of a Personalized RPG section. Third, the Pilot Study section provides the details of the pilot study, including research design, context and participants, data collection, and data analysis. The next section presents the results of pilot study. Interpretations of the results were then provided in the Discussion section. The article concludes with implications and limitations.

Cognitive load
Cognitive load theory indicated that information is processed in working memory, which is affected by intrinsic load (IL), extraneous load (EL), and germane load (GL) (Sweller et al., 1998). IL is the intrinsic nature of the learning materials and cannot be altered by instructional design (Sweller et al., 1998). When learning materials have high element interactivity, students are expected to have high IL to process several elements simultaneously (Sweller, 2010). Low-element interactivity materials do not expect high IL because those materials have fewer elements that can be processed serially rather than simultaneously (Sweller, 2020). EL refers to the load that does not contribute to learning (Sweller et al., 1998). It can be altered by instructional interventions and is determined by the instructional design (Sweller, 2010). If the instruction was poorly designed, high EL would occur among students. Thus, a good instructional design should be able to decrease EL (Sweller, 2020). GL is the invested cognitive effort that facilitates efficient learning (Sweller et al., 1998). If IL and EL were low, students could be directed to procedures that were relevant to learning (Sweller, 2010). Caution is needed not to exceed the limits of the total working memory (Sweller, 2020).
In gaming environment, cognitive expenditure is expected to be higher than traditional e-learning environment because gaming environment has rich multimedia that needs considerable amount of cognitive capacity to process the gaming environment when simultaneously interacting with game components (Mayer, 2010). However, research regarding cognitive load in gaming environment yielded mixed results. Chang et al. (2018) compared differences in cognitive load between gaming environment and traditional computer-based environment and students displayed lower cognitive load and better performance. Studies conducted by Huang (2011) and Schrader and Bastiaens (2012) found increased amount of cognitive load among students. The incorporation of PL into games will increase the complexity of the learning environment. How students' cognitive load would be affected in this complex personalized gaming environment needs more research.

Digital RPGs
RPGs are games that allow the players to assume the roles of fictional characters and operate those characters in a fictional game environment (Deterding & Zagal, 2018). Among different forms of RPGs, digital RPGs is the most used forms in educational settings. Digital RPGs are digitized tabletop RPGs that all the game character operations occur on a computer rather than the paper. Many RPGs features could be used for educational purposes. For instance, portraying game characters gives students the opportunities of controlling game characters, which could stimulate perspective-taking and experience-taking (known as immersion). Taking other people's perspectives allows players to practice social-emotional skills and deepen understandings of learning materials. Experiencing or immersing in a game story will assist players to adjust their behaviors in the real world.
Digital RPGs has been successfully implemented in various educational settings, such as engineering education (McConville et al., 2017), language learning (Ng et al., 2021;Peterson, 2016), and physical science (Garneli et al., 2019). Previous studies have proved RPGs' potential to be an effective approach of promoting knowledge acquisition and cognitive construction (Chen et al., 2021;Daniau, 2016;Liao et al., 2019;O'Brien et al., 2010;Yang & Quadir, 2018). For example, Kusuma et al. (2021) developed an RPG to support students' historical learning and results showed significant improvement on students' performance. Zhong (2022b) examined the effectiveness of an RPG and also found significant improvement in performance.

PL and RPGs
PL has been demonstrated as superior to traditional one-size-fits-all instructional approaches as it allows students to customize learning based on their own interests and abilities (Martin et al., 2020). Researchers have further indicated that PL is able to engage students in critical thinking and help them achieve higher level of learning (Zhang et al., 2020). Thus, PL has been applied to a variety of educational contexts and findings of those studies has supported the effectiveness of PL on students' learning outcomes, motivation, and metacognitive skills (Arroyo et al., 2014).
In recent years, researchers attempted to incorporate PL into educational games. From the experimental results, they found that personalized games have the potential to improve students' learning performance and reduce cognitive load Soflano et al., 2015;Yang & Quadir, 2018). For example, Zualkernan et al. (2010) used student's prior knowledge to determine the subsequent questions in an adaptive RPG environment. Their follow-up case study, using mixed method, showed positive influence on students' performance. Troussas et al. (2020) used a knowledge assessment module to assess students' knowledge level in the programming language Visual C# and generate personalized quiz questions based on each student's assessment result. Results of their study also yielded positive influence on students' performance. In Ku et al. (2016) study, students' cognitive preferences (Holists or Serialists) were utilized to personalize the content layout and navigation support. Results showed the personalized educational games was useful to enhance students' learning performance and reduce cognitive load. Students' facial emotions collected via webcam were used in Tsai et al. (2012) study to personalize the game difficulty and the learning materials difficulty. Increased motivation and satisfaction were reported in their study. Krouska et al. (2020) developed a personalized brain-based quiz game that was able to adapt the quiz content based on students' motivational state. Their study showed that students in the personalized learning group outperformed the non-personalized group. In the meanwhile, researchers pointed out the limitations of incorporating PL into digital RPGs. Plass and Pawar (2020) reviewed various implementation approaches and noticed that the variables considered for personalization were limited to cognitive and motivational variables (e.g., prior knowledge and motivation). This resulted in narrow approaches to personalize educational games. More learner attributes, especially from affective and sociocultural domains (e.g., emotional state), should be taken into consideration to personalize educational games.

SID model
Situational instructional design (SID) model was developed by Zhong and Xu (2019), aiming at addressing individual differences in instruction. SID model consists of two parts: learning readiness (LR) status and situational design model. LR status is the core concept of SID model that assists with identifying students' individual differences in recurrent skills, non-recurrent skills, and willingness (refer Zhong and Xu (2019) for definitions). Situational design model provides guidelines of designing instructional styles that match each LR status. Each instructional style is a combination of procedural learning activity, supportive activity, and relationship activity (refer Zhong and Xu (2019) for definitions).

Development of a personalized RPG
SID model was utilized in this study to guide the development of a personalized RPG because this model has consolidated both cognitive variables (prior knowledge and cognitive skills) and affective variables (emotional state) to design personalized learning. Additionally, SID model has sound theoretical basis that informs us how the learning environment should respond to students' differences along the identified variables. The development of the personalized RPG used RPG Maker MV and contained six steps: (1) identify recurrent and non-recurrent skills; (2) develop player diagnosis survey; (3) develop personalized responses; (4) determine game flow and interactions; (5) present game content; (6) test and launch.

Identify recurrent and non-recurrent skills
The first step is to identify related domain recurrent and non-recurrent skills. According to Zhong and Xu (2019), recurrent skills refer to students' proficiency in performing routine aspect of the problem, such as explaining the definitions and conducting related procedures. Non-recurrent skills represent students' ability in performing nonroutine aspect of the problem, such as evaluating, abstracting, and reasoning. In this study, students will be introduced to situational leadership theories and complete multiple case studies to practice applying theories. Related recurrent skills include understanding basic concepts of situational leadership (e.g., ability, willingness, and behavioral indicators of ability and willingness), defining the four leadership styles, and describing how each leadership style looks like. Examples of non-recurrent skills include assessing performance readiness level, identifying matching leadership style, and explaining the results of mismatch between readiness and leadership style in the specific case context. Recurrent and non-recurrent skills identified in this study are summarized in Table 1.

Develop player diagnosis survey
Player diagnosis survey is used to identify each player's LR status. In this study, we adopted the LR survey used in Zhong and Xu's (2019) study (validated by Delahaye & Smith, 1995; Cronbach's alpha reliability = .73). The six eight-point Likert survey questions were slightly revised to cover the topic in this class (see "Appendix 1" for the player diagnosis survey). The first two questions were to identify students' recurrent skills and marked as r score. The third and fourth questions were to evaluate students' non-recurrent skills and marked as nr score. The last two questions were to assess students' willingness and marked as w score. Each score is the sum of the responses to the two questions, ranging from 1 to 16. If falling between 1 and 8, this score would be stored as 0 in the game. If falling between 9 and 16, this score would be stored as 1 in the game. The LR survey was implemented in the game via an NPC when players enter the game (see Fig. 1). Players could not skip the LR questionnaire as it was required to move forward. The matching map and classification of LR status and instruction is provided in Table 2.

Develop personalized responses
Development of personalized responses is to generate appropriate instructional style for each LR status. In this study, a total of ten case study materials, which were course materials (details of each case study can be found at "Appendix 2"), were utilized to develop the eight instructional styles as described in Table 4. Case study one, two, and three aimed to develop recurrent skills. Case study four, five, and six were to develop non-recurrent skills. Case study seven, eight, nine, and ten focused on applying the theories to solve problems. Relationship activities were provided to the instructor as a separate instruction guide (see "Appendix 3" for a sample guide, adopted from Zhong, 2022c). Examples of high relationship activities included explaining why, emphasizing how to, sharing the responsibility of decision-making, and encouraging questions. Examples of low relationship activities included keeping the emotional level in check, directly explaining specific facts, and encouraging autonomy and freedom for risktaking. Explanations of instructional styles are summarized in Table 3.

Determine game flow and interactions
Game flow depicted the overall flow of the game, from the moment it's launched to the end of the game. It visually shows how the game works and what game players will experience. Figure 4 shows the flow of the personalized RPG in this study. As shown in Fig. 2, players could access the game via different interfaces, such as mobile devices, laptop, and desktop. After launching the game, players would be directed to player diagnosis component.

Present game content
Game content presentation is to present the personalized responses. Each response consisted of four case studies and was presented as four maps in the game. Interactions were implemented via different NPCs in the maps. Figure 3 is an example of a map presented to the player. The player was directed the first map and interacted with the NPCs in the map once s/he completed the LR survey. When the first map Table 3 Explanations of instructional styles (adopted from Zhong, 2022c)

Instructional style Explanation
was completed, the player could use the doors to navigate to the next map where s/ he would complete the next case study. This process would repeat until all four maps were successfully completed. The player would be directed by an NPC to the exit map where s/he would end the game (see Fig. 4).

Test and launch
Test and launch are to identify defects and bugs in the game to improve stability and performance. In this study, the game was finally launched as a web-based game (https:// bessi ezhon glin. itch. io/ listo-system-leade rship) and a pilot study was conducted to (a) test the quality of the game and (b) examine its effectiveness on students' learning performance and cognitive load. The following section provided detailed descriptions of the pilot study. Pilot study

Research design
The pilot study utilized one-group repeated measurement design to investigate the effect of personalized RPG on students' performance and cognitive load. The rationale is that only one group of participants were available to the study (Gall et al., 2007). Before the intervention, participants' academic performance was assessed by a pretest and cognitive load was assessed by a survey to establish the baselines. After the intervention, participants' academic performance was assessed again using a posttest, which was slightly modified from pretest. Cognitive load was assessed again using the same tool. Differences before and after the intervention were calculated and compared to identify the effectiveness.

Context and participants
This study was conducted in a workforce education program at a large university in the midwestern United States. The game was implemented in the fifth learning module of a 16-week online course. This module would introduce situational leadership theories to students. The instructor of the course participated in the development of the module. On the first day of the class, the recruitment letter was emailed to all students who enrolled in this course. A consent form that explained the details of this study was attached in the recruitment email. Forty-one students signed consent forms and indicated their participations in the research, including six males and twenty-two females. All participants were between 24 and 35 years old.

Data collection
Participant's academic performance was evaluated by a pre-/posttest of their knowledge of the respective content. The two tests/exams were created by the faculty who had previously taught the content. The two tests/exams were revised and refined by specialists with expertise in assessment and loaded into the learning management system. Full score of each test/exam was 100 points. Pretest was delivered in week four, the week before the intervention. Posttest was delivered in week six, the week after the intervention. Participant's cognitive load was measured by a survey developed and validated by Leppink et al. (2013). This survey consisted of nine Likert scale questions that measured three types of cognitive load devoted in the learning process. This survey was also delivered to participants before and after the intervention.

Data analysis
Participants' performance on the pre/posttest was graded by two trained raters independently and average scores between the two raters were used for analysis. Interrater reliability analysis using Cohen's Kappa was conducted to determine the internal consistency among the raters. Crobach's Alpha was conducted to determine reliability of the cognitive load survey. Shapiro-Wilk test showed that the data significantly deviated from a normal distribution (F = .95, p = .011 < .05). Thus, Wilconxon signed-rank test was performed to determine the impact of the personalized RPG on participant's academic performance and cognitive load. Spearman' rank-order test was conducted to determine the relationship between learning performance and cognitive load.

Results
The interrater reliability for the raters on pre/posttest was found to be Kappa = .87 > .75, 95% CI (.55-1.00). The cognitive load survey was found to be reliable (Cronbach's α = .897). Demographic characteristics of the participants are summarized in Table 4. Descriptive results for learning performance and cognitive load are summarized in Table 5.   To determine the impact on learning performance and cognitive load, Wilconxon signed-rank test was conducted. Results (see Table 6) showed significant difference in learning performance (Z = 5.582, p < .001), EL (Z = 5.655, p < .001), and GL (Z = 5.654, p < .001) but not in IL (Z = 1.897, p = .058 > .05).
To determine the relationship between learning performance and cognitive load, Spearman's rank-order test was conducted. Results (see Table 7) showed that there was a negative weak correlation between performance and IL (r s = − .349, p = .025 < .05) and a negative moderate correlation between performance and EL (r s = − .423, p = .006 < .05). There was a positive strong correlation between IL and EL (r s = .644, p < .001).

Discussion
This study developed a personalized RPG via SID model and piloted its effectiveness on students' performance and cognitive load. Results showed that students' performance has been significantly improved after trained in the personalized RPG environment. This study also observed significantly decreased EL and increased GL. These findings indicated that personalized RPG environment is effective in improving learning performance, reducing EL, and promoting GL. This is consistent with Chang et al. (2017), Toth and Kayler's (2015), Hwang et al. (2012), Zualkernan et al. (2010), and Zhong's (2022d) studies, which observed significant improvement on learning performance and cognitive load among students who were trained in the personalized gaming environment.
Additionally, this study found a negative moderate correlation between performance and EL and a positive strong correlation between IL and EL. Performance was correlated to IL, but the correlation was negative weak. Correlation between GL and other variables was not found in this study. These findings imply that although decreased EL could leave students more GL capacity for efficient learning, this would not necessarily lead to performance improvement. Students need to be motivated to invest sufficient GL to actively process the learning materials and thus, improve performance. In this study, students' EL decreased and GL increased but correlation between EL and GL was not observed, indicating that the personalized RPG environment constructed in this study has the potential of motivating students and promote sufficient GL investment in efficient learning. This echoes previous empirical studies (e.g., Fadda et al., 2022;Westera, 2019;Yu et al., 2021) as well as related theoretical discussions regarding motivational effects on cognitive load proposed by Paas et al. (2005). These findings also align with the cognitive load theory (Kalyuga, 2009;Sweller, 2010Sweller, , 2020 that IL is determined by the learning content and cannot be altered by instructional design. EL and GL can be changed by instructional interventions and decreased EL would improve students' performance.

Implications and limitations
Findings of this study have three implications. First, SID model could be an effective approach to include cognitive and affective variables when incorporating personalization into educational games. SID model is a practical approach that not only provides approaches to identify students' cognitive and affective differences but also supplies recommendations on how the learning environment should respond to students' differences via different instructional styles. Second, educational game designers need to consider not only reducing EL but also promoting GL investment in the gaming environment. Reducing EL could increase GL capacity but not the actual GL investment. Third, motivational design is necessary in game design to direct students to procedures that are relevant to learning and exert sufficient GL. Keller's (1983) ARCS (Attention, Relevance, Confidence, Satisfaction) model is a good approach to incorporate motivational components in educational games. This study implies that motivation theories or models need to be included in current design models to promote efficient cognitive processing in educational games. This study also has two limitations. First, control group was not included in this study so it's difficult to determine what game features have contributed to students' improvement. Future studies are suggested to include a control group to study how game features impacted students' performance. Second, participants' demographic characteristics limit the generalization of this study. Most participants were female students who had limited gameplay experience. Other populations, such as male students who had more gameplay experience may product different results. Future research is recommended to study more diverse student populations.

Conclusion
This study developed a personalized RPG via SID model and piloted its effectiveness on students' performance and cognitive load. Results of the pilot study demonstrated the effectiveness of the personalized RPG on students' performance and cognitive load. This study also found that increased GL capacity would not necessarily lead to performance improvement. Students need to be motivated to invest sufficient GL to actively process the learning materials, and thus, improve performance. Findings of this study revealed the important roles that motivation plays in facilitating efficient learning in gaming environment. Researchers and practitioners are suggested to use findings of this study to guide future game designs. Case seven Case seven describes behavioral interactions between Kelly Fontane (a training specialist for new office staff training) and Julene Garfield (an employee in Kelly's team) regarding encoding tasks and depositing forms. Students will assess Julene's Performance Readiness level for the task of encoding. Students will also assess Kelly's leadership style and the reasons for Kelly to use this particular leadership style

Appendix 1: Player diagnosis survey
Case eight Case eight describes Yuki Tanaka's (supervisor of the sales representative team at Listo Systems Company) behavioral interactions with her team members when dealing with performance problems in the team. Students will identify and assess Yuki's actions that indicate task behavior and relationship behavior. Based on the assessment results, students will identify Yuki's leadership style and assess the appropriateness of this leadership style Case nine Case nine describes Raul Martinez's, (a graphic design supervisor for Listo Systems Company) behavioral interactions with his group regarding the production in the graphic design department. Students will identify the group's Performance Readiness level for the task of production in the graphic design department and leadership style that best matches the group's Performance Readiness level. Students will also assess leadership style that Raul is using and the indicators that Raul is using the appropriate role. Additionally, students will explain potential hindering roles that Raul could be using if there is a performance readiness/style mismatch Case ten Case ten describes Michelle Hoffman's (supervisor of the market research group) behavioral interactions with her team when doing market research. Students will identify the group's Performance Readiness level for the task of doing market research and leadership style that best matches the group's Performance Readiness level. Students will also assess leadership style that Michelle is using and explain the group's likely response to the mismatch of leadership style

Appendix 3: Sample relationship activity instruction guide
For R1 students, the instructor is suggested to provide direct explanations of the game tasks; provide game task information in digestible amounts; help the student step by step and avoid overwhelming; instruction focuses on task completion; reinforce small improvements; explain consequences of nonperformance, such as not completing the game; check emotional level regularly. For R2 students, the instructor is suggested to explain consequences of nonperformance, such as not completing the game; encourage trying; support risk-taking; praise and build confidence; ask students question to clarify their understandings of the game tasks; discuss details of the game tasks; explore related non-recurrent skills; compliment students when they finish the game tasks.
For R3 students, the instructor is suggested to provide direct explanations of the game task; support risk-taking; praise and build confidence; discuss details of game tasks; ask students question to clarify their understandings of the game tasks; encourage students to ask questions; compliment students when they finish the game tasks; For R4 students, the instructor is suggested to explain consequences of nonperformance, such as not completing the game; seek "buy-in" through persuading; discuss details of game tasks with students; praise and build confidence; compliment students when they finish the game tasks.
For R5 students, the instructor is suggested to provide game task information in digestible amounts; help the student step by step; ask students question to clarify their understandings of the game tasks; discuss details of the game tasks; encourage students to ask questions; explore related non-recurrent skills.
For R6 students, the instructor is suggested to ask students question to clarify their understandings of the game tasks; discuss details of the game tasks; explore related nonrecurrent skills; reinforce small improvements.
For R7 students, the instructor is suggested to provide direct explanations of the game tasks; help the student step by step; instruction focuses on task completion; reinforce small improvements.
For R8 students, the instructor is suggested to monitor gameplay activities; provide relatively light supervision regarding game completion; give freedom for risk taking; encourage autonomy of gameplay, such as explore other maps in the game.

RPGs
Role-playing games PL Personalized learning IL Intrinsic load EL Extraneous load GL Germane load SID Situational instructional design LR Learning readiness