Static and dynamic eye movement metrics for students’ performance assessment

Abstract

This paper reports on the feasibility of using eye tracking as a tool for students’ performance assessment in a medical serious game. We are particularly interested in analyzing the relationship between learners’ visual behaviour and their performance while solving medical cases. The objective of this study is twofold. First, we analyze how the students visually explore the learning environment across different areas of interest. Second, we examine whether static and dynamic eye tracking metrics can have an impact on students’ reasoning performance. Results revealed statistically significant associations between eye movement metrics and students’ outcomes. Particularly dynamic metrics better reflected students’ analytical reasoning abilities. Our findings have implications for the educational technology community seeking to gain a deeper understanding of the students’ learning experience.

Introduction

In the recent years, there has been an increased interest in tracking users’ eye movements and particularly in assessing what can gaze data reveal about the users’ experience. In fact, eye-tracking technologies attempt to make the invisible visible. Many researchers in various domains have employed eye tracking as a technique to capture where ocular attention is focused on a visual display at any specific time, as well as the sequence of fixations followed from one area of interest to another.

Eye tracking is used in human-computer interaction to study usability issues (Eraslan et al. 2015; Poole, 2005; Weill-Tessier et al. 2016) and in visualization tasks to provide effective user-adaptive systems (Jaques et al. 2014; Lallé et al. 2017; Toker & Conati, 2014). The development of computer-based adaptive systems in learning environments using eye tracking relies principally on the assessment of users’ visual behaviour, users’ visual pattern and users’ learning. Indeed, enhancing students’ learning outcomes and facilitating knowledge construction and acquisition is of primary interest. Eye tracking technique has proven its efficiency in educational environments to understand how students evolve through the learning process using different eye gaze metrics (Ben Khedher et al. 2017; Lallé et al. 2017; Luo et al. 2017; Raptis et al., 2017; Trevors et al. 2016; Wang et al. 2016).

The majority of eye tracking studies in the literature use static metrics such as fixation count, fixation duration or saccade length (Ho et al. 2014; Kardan & Conati, 2012). Yet, some researchers are concerned with the sequential properties of eye movements such as the visual scan path (Lee & Wu, 2017; Susac et al. 2014). These dynamic metrics provide reliable information about moment-to-moment changes in the participants’ visual behaviour dynamics. This paper proposes to use both types of eye gaze metrics, in order to assess learners’ performance as they interact with a medical serious game. We explore the following two research questions. RQ1: What can fixation-based static metrics reveal about students’ experience within the learning environment? RQ2: Do students’ visual patterns across areas of interest (AOIs) have an impact on their performance?

The rest of the paper is structured as follows. Section “Related Work” presents related work. Section “Experimental Design” describes our learning environment and the experimental protocol. Section “Results and Discussion” discusses the obtained results. Section “Conclusion” concludes and presents directions for future work.

Related work

The aim of the educational technology communities is to constantly seek to offer a pleasant and effective learning environment in order to keep students immersed and foster learning performance (Jraidi et al. 2014; Jraidi & Frasson, 2010; Sawyer et al. 2017). Eye tracking can provide valuable insight into the learners’ gaze behaviour during cognitive task resolution using static and dynamic metrics (Takeuchi & Habuchi, 2007). Static metrics such as fixation duration, fixation count or saccade length are measured from the observed data according to a temporal dimension. They are used to assess where visual attention is focused (Toker et al. 2013). Fixation is when the eyes momentarily keep fixating a particular target for a certain duration. Saccade corresponds to a rapid eye movement from one fixation to another (Holland & Komogortsev, 2011; Yoshimura et al. 2015). Dynamic metrics such as scan path are measured according to spatial-temporal dimensions. Scan path is a series of fixations and saccades representing the visual trajectory of a student’s eye movements across the AOIs (Blascheck et al., 2014; Duchowski et al., 2010). Dynamic metrics are used to analyze the students’ visual shifts along the time and across different areas of interest. However, a main issue is to relate these gaze metrics to students’ learning.

Pachman and his colleagues (2016) used fixation-based metrics to detect learners’ confusion in digital learning environments. They found that confusion ratings were positively correlated with the fixation duration on non-relevant areas, which helped them detect potential confusion areas with the future goal of setting self-regulatory techniques for students to manage their confusion during learning (Pachman et al. 2016). In (Ben Khedher & Frasson, 2016), the authors used eye movements in order to predict learners’ level of performance during their interaction with a narrative-centred learning game. They examined the relationship between static metrics such as mean fixation duration and learners’ outcomes. They were able to discriminate between two groups of learners in term of scores (high and low) in different multiple-choice quizzes. Lum et al. (2016) investigated the relationship between the number of fixations and learners’ mental workload during the completion of a computer-based training program. The results showed that there were statistically significant correlations implying that when the task was not challenging enough, learners’ visual attention decreased (Lum et al. 2016).

Despite that static metrics are being used as a main measure for most studies, they do not glean enough information to show that focusing on a particular area for a certain period of time is the cause of performance differences among learners. Hence the use of dynamic metrics, which provide a deeper understanding of the learners’ visual behaviour. In fact, scan paths are useful tools for providing differences in individuals eye movement patterns in the context of visual learning (Eisenbarth & Alpers, 2011; Hyönä, 2010; Krupinski et al., 2006; Xu et al., 2018). For instance, in the work of Byun et al. (2014), the authors analyzed learners’ in-situ data to investigate how eye tracking can be used to evaluate students’ performance while they are interacting with a serious game. Scan path dynamic metrics were used to differentiate between novice and expert players. Results showed that experts have a more organized visual path and tend to focus more on the relevant areas in the game compared to the novices who have a randomly visual trajectory (Byun et al. 2014). In the same context, Susac et al. (2014) also used scan path measures to discriminate between expert and non-expert learners while solving mathematical equations. Results indicated that expert students executed a well-defined and organized visual strategy as compared to non-expert students who focused more on the presented solutions (Susac et al. 2014).

In this paper, we propose to study both static and dynamic fixation-based metrics to assess students’ performance within problem-solving tasks. Our objective is to use both metrics to monitor the analytical reasoning process of novice medical students as they interact with a learning environment and explore a series of observations to resolve some clinical cases. First, we seek to analyze the students’ visual behaviour across different areas of interest and check whether there are particular areas that contribute to the students’ success. Second, we aim to examine whether static and dynamic eye gaze metrics can have an impact on students’ reasoning outcomes.

Experimental design

An experimental study was conducted to track students’ visual behaviour while interacting with a learning environment called Amnesia. Amnesia is a medical serious game developed to assess novice undergraduate medical students’ cognitive abilities through clinical problem-solving tasks. The design of the environment was examined and validated by a medical professional. The game features a virtual hospital where the user is playing the role of a doctor who is mistakenly diagnosed with amnesia and trapped within this hospital. The players need to prove that they do not suffer from amnesia by resolving first some cognitive tasks such as simple visual memory exercises and logic tests (e.g. number series and analogies). Then, they have to demonstrate their clinical skills by resolving six medical cases.

For each case, the students are instructed to identify the correct diagnosis and the appropriate treatment. For that purpose, they have to consider a series of observations such as patients’ demographic information, symptoms and antecedents. The players can make initial hypotheses from the demographic information and the symptoms exposed by the patient and establish a first diagnosis. The different diseases they have to discover are respectively: flu, bacterial pneumonia, measles, Ebola, mumps and whooping cough. For each diagnosis, different response alternatives are given and students have three attempts to find out the correct response. They can also collect additional clinical data such as analyses and antecedents until reaching the right diagnosis. Once the diagnosis is established, the students have also three attempts to identify the adequate treatment and after three errors made either in the diagnosis or the treatment, the game is over. Between the three attempts, the students can adjust their hypothesis and correct their answer.

Protocol

Upon arrival at the experimental room, students were asked to sign an ethic form explaining the study and the material. Then, they were placed in front of the eye tracker and were informed that free head movements were allowed. Following the calibration process, the game was displayed. An introductory scene informs the players about how to use the system and the main objectives of the learning environment. During the session, guidance messages were shown to help them move forward in the hospital. After 30–45 min of interaction, the session ends. Participants were then asked to fill in a post-game questionnaire about the game design and usability in order to collect feedbacks for further improvement.

Participants and apparatus

Fifteen undergraduate medical students were recruited for the experiment. Their eye movements were recorded using a Tobii Tx300 eye tracker at a sampling rate of 300 Hz. Infrared sensors and a camera were used to record the session on a 23-in. computer monitor with a (1920 × 1080) resolution. The Amnesia environment was presented on this eye tracking system, which was approximately placed at a distance of 65 cm from the participant. A nine-point calibration grid was used to evaluate the eye gaze quality of each participant. The calibration process consists of evaluating the quality of the measured gaze points. It is a mandatory step in eye tracking in order to keep in the study only reliable data. All the participants who took part in the study successfully passed the calibration step (see Table 1 for participants’ statistics).

Table 1 Participants’ statistics

In order to obtain a detailed analysis of the students’ eye movements, specific areas of interest (AOIs) representing task-relevant regions of the screen were created in each medical case (see Fig. 1). Six AOIs were defined as follows: Information (I), Antecedents (A), Symptoms (S) Analysis (N), Diagnosis (D) and Treatment (T). The I area of interest includes the demographic information of the patient (e.g. first and last name, age, weight and height). In the antecedent AOI, we introduce the eventual allergies and prior diseases that the patient could have. We present in the S region all the symptoms related to the specific disease. The analysis AOI includes data on the temperature, heart rate and the blood pressure of the patient. Concerning the diagnosis area, it shows different suggestions of answers among which the participant may select one. Finally, the T area of interest presents different suggestions for the potential treatments of the selected disease. All the medical cases have the same AOIs. Gaze data relative to each area were recorded. From the obtained data three metrics were computed: two static metrics namely fixation duration and time to first fixation, and a dynamic metric namely scan path. Fixation duration is the average time spent on a particular AOI. Time to first fixation indicates the amount of time it takes to look at a specific AOI from stimulus onset. Scan path is a series of fixations and saccades representing the visual trajectory of a student’s eye movements across the AOIs.

Fig. 1

The six areas of interest identified in all the medical cases

Results and discussion

In this study, we had two main objectives. First, we aimed to track students’ eye movements while they were resolving medical cases. We wanted to have a general overview of their visual behaviour across the different areas of interest in each medical case. The goal was to identify which areas of interest caught the most the students’ attention and whether such area(s) contributed to the students’ success using static metrics. Second, we sought to investigate students’ visual patterns using the scan path dynamic metric. That is, can the scan path actually reveal the students’ thought process? Can this metric accurately reflect the learners’ performance?

Fixation-based metrics and students’ performance assessment across the AOIs

In each medical case, participants have the ability to inspect different regions of the screen as they attempted to make a diagnosis based on Symptoms (S), Information (I), Antecedents (A) and Analyses (N) as shown in Fig. 2. In each region, the fixation metrics were recorded to find out the most visited areas and to identify whether they could possibly lead to successful responses.

Fig. 2

A screenshot of one medical case. The red colour represents the most fixated areas of interest within the screen, followed by yellow and then green for the areas with less fixation intensity

How students explore the learning environment?

A preliminary statistical comparison between the different areas of interest using the mean values (M) and the standard deviations (SD) was performed. The results showed that the Symptoms’ area was by far the most fixated zone by all the participants in all cases in terms of fixation duration. The most significant fixation rate was found in the last case (M = 28.72, SD = 17.48) and the least was during the resolution of case 1 (M = 13.90 SD = 11.48). Indeed, in the first case, participants discovered the interface for the first time. For this reason, the time allocated for the symptoms was not as different as the other areas, as compared for example to the analysis area of interest (M = 10.37, SD = 7.05).

In accordance with these findings, a one-way ANOVA was performed to examine whether there were significant differences among all the AOIs in term of fixation duration. Table 2 shows statistically significant results (p < 0.001) among all cases. Tukey post-hoc tests were conducted where all areas of interest were compared by pairs in order to find out which area of interest attracted the most the students’ interest. The results showed that the Symptoms’ AOI differed significantly (p < 0.05) from the other areas in almost all cases. Based on fixation duration, the time spent looking at the Symptoms’ area far exceeded the time dedicated to the other areas (p < .001). As a matter of fact, these results are obvious since the analysis of a patient’s symptoms is a paramount step in any clinical diagnosis process.

Table 2 Descriptive statistics for the fixation duration metric across all the AOIs, Means (Standard Deviations)

Do fixation-based metrics have an impact on students’ responses?

In this study, several MANOVAs were performed to check in each case, whether looking to the “S” AOI can predict students’ responses and if not, which area can then better predict it.

Symptoms’ area

The results of the MANOVAs are shown in Table 3. For the first case, the analysis was not conducted since the number of subjects was not fair between the two groups (group 1: success; group 2: failure). For cases 2, 3, 4 and 5, the results did not show a statistically significant relationship between the fixation duration and the identification of the correct diagnosis. Finally, for the last case, a significant effect was found showing that focusing on the “S” area may potentially have an impact on the success or failure.

Table 3 MANOVAs’ results summary between the symptoms’ fixation duration and the diagnosis success/failure

Remaining areas

In the previous section, the results demonstrated that one cannot conclude that longer fixation on the Symptoms’ AOI impacts the students’ performance. In this section we present additional case-by-case MANOVAs to investigate whether there is such area that leads to the students’ success. Both fixation duration and time to first fixation were analyzed.

Case 1 was discarded since all participants succeeded in identifying the diagnosis. Case 2 (F (1, 8) = 4.946) and case 5 (F (1, 3) = 0.084) showed no significant differences (p = n.s) between the areas of interest in terms of both fixation metrics. For the remaining three cases, the results were statistically significant in terms of time to first fixation. Post hoc tests were performed for separate correlational analyses to check which AOI could potentially abet students’ success. Results showed that there was not a unique AOI that is related to participants’ performance, but in each case there were different areas. For case 3 and 6 a significant effect was found for all areas (p < 0.05). In case 4, a unique significant result was found for the Antecedents’ region (p < 0.001). In fact in this case, the diagnosis the students should identify is Ebola, and they needed to pay particular attention to what was listed in the antecedents’ area, which included an important clue (″recently travelled to Guinea″) that may help students give the correct answer.

In summary, we were able to identify a relationship between some areas of interest and students’ performance using fixation-based static metrics. However, we cannot state that these static metrics can reliably assess students’ performance. For instance, as stated in different studies (Djamasbi, 2014; Poole & Ball, 2006), longer fixations do not always reflect users’ engagement; fixation metrics and learners’ performance may be negatively correlated.

Performance assessment using scan path metrics

In this section, we investigate the relationship between students’ saccadic eye movement sequences and their performance. We use students’ scan paths and a sequence alignment algorithm namely, Smith-Waterman, to evaluate learners’ analytical reasoning. Smith-Waterman algorithm is based on dynamic programming for pairwise sequence comparison (Setubal et al. 1997). Our objective is to use this method to discriminate between the learners in a way that identifies those who follow a correct analytical reasoning process using students’ visual scan paths. The alignment process is divided into three parts.

First, students’ scan paths are recoded as a sequence of characters representing the student’s followed steps to solve each case (e.g. ″IIIAASSSSSNNNNSSNDDDSSDDTTDTT″). Each character of the sequence is a visited AOI with a fixation duration above a threshold of 250 ms. Redundant characters are collapsed to retain a unique occurrence for each visited area in the scan path (e.g. ″IASSNSNDSDTDT″).

Second, a reference sequence is used to represent the hypothetico-deductive clinical reasoning process. This target sequence is a sequential representation of the different steps a novice student has to perform to accomplish an optimal reasoning process yielding to this following sequence: ″ISANDT″. Indeed, in this analytical process, the clinician starts by collecting the patient’s information, then he can make initial hypotheses from the symptoms exposed by the patient. Additional clinical data such as analyses or medical antecedents are then collected to validate or discard his hypotheses until he reaches the correct diagnosis and treatment. In order to have deeper information about the hypothetico-deductive clinical reasoning process, please refer to (Nendaz et al. 2005).

Third, both obtained sequences (i.e. the student’s scan path and the reference sequence) are compared using the local sequence alignment Smith-Waterman algorithm. Aligning two sequences using the Smith-Waterman algorithm implies identifying regions of similarity between both sequences. The first step of the Smith-Waterman algorithm is to create a similarity-based matrix where the first column and row are initialized with zeros. The remainder cells are filled using the local alignment recursion relation. Once the matrix is completely filled, the algorithm selects the cell with the highest score. Then, from that cell, it backtracks through the array until reaching the first zero. The constructed path represents one of the possible optimal local alignments for the two sequences.

In order to quantify the optimal alignment found, a final score is computed based on three scoring metrics namely; match, mismatch and gap. A match means that two characters in both sequences are identical, a mismatch requires a mutation operation that is substituting one character to another, and a gap implies an insertion or a deletion in one of the two sequences. The higher the computed alignment score, the more the two sequences are similar. We refer to (Ben Khedher et al. 2018) for more details about this procedure.

In order to highlight the contribution of the scan path dynamic metric in assessing students’ performance, we are interested, in the next section, in analyzing the relationship between the alignment metrics and participants’ progression through the three attempts of the tasks’ resolution. Table 4 presents descriptive statistics about the different response attempts in each medical case: the Average Exposure Time (A.E.T) in seconds represents the average of the time spent solving the case, the Total Attempts is equal to the total number of attempts made per case and the Total Error Attempts shows the total number of failed attempts. According to these statistics, the first case is where the students spent more time to resolve as well as the highest number of attempts because it is the first medical case in the game. In fact, the students were not familiar with the environment, it takes time to understand the interface they were shown. During the first case exposure time, almost half of the time is spent exploring the environment and reading the different symptoms and analysis (A.E.T between the beginning of the case and the click on the diagnostic button = 699 s instead of 258 s in the third case).

Table 4 Descriptive statistics for the response attempts in each medical case

Performance assessment across the different attempts

A one-way ANOVA was conducted in order to analyze the alignment metrics (score, gap, match and mismatch) across the three attempts allowed to resolve each medical case.

First, a significant contribution was found across the three attempts in terms of alignment score (F (3, 52) = 5.193, p < 0.05). This indicates that at each failure in one attempt, the alignment score decreases which suggests a reasoning sequence that deviates from the reference sequence. Similarly for the number of gaps and mismatches, statistically significant differences were reported respectively (F (3, 52) = 3.644, p < 0.05) and (F (3, 52) = 24.057, p < 0.001). These results point out that the alignment metrics are strongly linked to students’ performance when reasoning.

We performed a scheffe post hoc test to present detailed group-based statistics for each alignment metric. Results indicated that the alignment score significantly decreases in the third attempt compared to the first and second ones as shown in Fig. 3. These findings clearly show that after two failures, the student cannot produce effective results anymore, especially if he did not yet found the right diagnosis. In terms of sequence alignment, it can be said that the similarity between the reference sequence and the students’ reasoning sequence tend to decrease as the number of attempts increases since the number of gaps and mismatches has almost doubled in the third attempt compared to the first one. In fact, a low alignment score means a low number of matches and a high number of gaps.

Fig. 3

Comparison of alignment metrics for the three different attempts. a Score b Gap c Mismatch

To sum up, two methods that assessed the performance of the students as they were resolving medical cases were presented. In the first approach, we used static eye gaze metrics namely, fixation duration and time to first fixation in order to investigate how the participants visually explore the learning environment and specifically what can these metrics reveal about students’ visual behaviour. The analysis demonstrated that almost all students have a similar visual pattern revealing that the symptoms’ area of interest was fixated the most. Yet, contradictory results have been found showing that focusing on that region has no effect on learners’ outcomes. In the second approach, we showed how to model students’ eye movements using the dynamic eye gaze metric (i.e. scan path) and differentiate between them in a way to identify the learners who executed correct vs. incorrect reasoning process. Through these experiments, we were able to answer our two research questions by highlighting the importance of using scan paths when analyzing students’ eye movements since they enable to monitor moment-to-moment changes in the participants’ visual attention.

Conclusion

In this paper, we proposed a two-fold approach to assess students’ reasoning process using first static metrics namely, fixation duration and time to first fixation and second a dynamic metric namely, scan path. An experimental protocol was established by recruiting 15 medical students to record their eye movements. Participants interacted with a medical serious game designed to elicit their analytical reasoning abilities when solving six clinical cases. Results showed that using static metrics, we were able to have a general overview of participants’ visual experience within the learning environment. On one hand, we observed that the symptoms area of interest was by far the most fixated area regarding its importance in the reasoning process. However, on the other hand, fixating this area for a long period of time does not always lead to good performance. Then, we used the scan path metric in order to assess students’ analytical abilities. This enabled us to discriminate between students in a way that identifies those who performed a correct/incorrect reasoning during medical case resolution.

These findings have potential implications particularly in high-risk medical environments where it is paramount to constantly monitor students’ performances. As future works, we aim to propose a multimodal approach to assess students’ reasoning performance on the basis of their cerebral and visual behaviour using jointly electroencephalography (EEG) and eye tracking. In the short term, we are planning to study two EEG indices namely, engagement and workload in order to monitor participants’ brain activity through the different game phases to investigate whether there are differences in terms of performance. In the long term, we will be interested in developing help strategies that will be used in real time during the interaction with the environment to provide support for the students with the goal of improving their reasoning process.