Table 3 An overview of the dataset in the selected articles to address RQ2

(USTC-NVIE) Wang et al. (2010)

ER

S &P

3-4 mins emotional and 1-2 mins neutral videos from internet

215 healthy students (157 M and 58 F)

236 apex images. Visible and thermal.

5 EAs & self-report.

6 basic emotions (happiness, sadness, surprise, fear, anger, and disgust), average arousal and valence.

Yes\(^{{a}}\)

Cocea et al. (2011)

E

WE

An online course (HTML-tutor) in 7 sessinos

48 users

14 logged events

3 EA

Engaged, disengaged, or neutral

N/A

AlZoubi et al. (2012)

E

S

An intelligent tutoring system with conversational dialogues (AutoTutor) in 45 mins learning session

27 students

20-second interval of biosensor signals

Retrospective self-report

8 affective states: boredom, confusion, curiosity, delight, flow/engagement, surprise, and neutral.

N/A

S-Syun et al. (2012)

ER

S

An intellingent robot platform (MERO): greeting, identification, and a questions game.

10 participants

1,400 elements

(humans numerical values)

4 facial expression states: smiling, surprised, neutral and angry

N/A

Whitehill et al. (2014)

E

S

A cognitive skills training software .

34 undergraduate students

10 seconds videos

7EA

Engaged, Not Engaged ([Very engaged,Engaged],[Nominally engaged,Not Engaged])

N/A

Schiavo et al. (2014)

E

S

A video game: single player level, “Operation 40” of “Call of Duty - Black Ops” video game.

22 participants (3 F, 19 M)

12420 samples

self-annotate using ExperienceSampling Method (ESM) (Larson and Csikszentmihalyi 2014)

Neutral, Engaged, Stress

N/A

Woo-Han Yun et al. (2015)

E

S

A testing interactive software.

12 Children

2,745 of 30 second video clips

1 EA

4 engagement levels: high/low interest, low/high boredom

N/A

(DAiSEE) Gupta et al. (2016)

E

W

2 videos (educational and recreational)

112 students (32 F and 64 M)

9,068 clips (10 secs)

10 EA

4 levels of 4 affective states: engagement, frustration, confusion, boredom

Yes\(^{b}\)

Zaletelj et al. (2017)

E

S

4 lecturing sessions (@25-min) in offline classroom setting

18 students

videos and kinect features

5 EA

3-level scale attention score (high, medium, low)

N/A

Monkaresi et al. (2017)

E

S

Writing task (draft-feedback-review)

22 students

1,325 video segments

Concurrent and retrospective self-report

Not engaged, engaged

N/A

(UE-HRI) Youssef et al. (2017)

E

S

Interaction with Pepper robot

195 participants (125 M, 70 F)

Sign of Engagement Decrease (SED), Early sign of future engagement BreakDown (EBD), engagement BreakDown (BD), Temporary disengagement (TD)

Yes\(^{c}\)

(BAUM-1) Zhalehpour et al. (2017)

ER

S &P

Short video clips

31 subjects (Turkish)

1,184 clips

5 EA

13 emotional and mental states, which are Anger (An), Disgust (Di), Fear (Fe), Happiness (Ha), Sadness(Sa), Surprise (Su), Boredom (Bo), Contempt (Co), Unsure (Un), Neutral (Ne), Thinking (Th), Concentrating (Con), Bothered (Bot)

Yes\(^{d}\)

Hussain et al. (2018)

E

WE

Social science course on virtual learning environment (VLE)

383 students

Log file

N/A

IF (score on the assessment>= 90) OR (final results=Pass AND total number of clicks>- average clicks), then label = high engagement. Otherwise -> Low engagement

N/A

Psaltis et al. (2018)

ER

S &P

prosocial games: Path of Trust (original version and stripped-down version)

72 participants

750 videos from 15 subjects (3 seconds)

Retrospective self-reports

5 basic emotions (anger, fear, happines, sadness, surprise)

Yes\(^{e}\)

Rudovic et al. (2018b)

E

S

25 min therapy session with NAO robot to learn four basic emotions: sadness, fear, anger, and happines

35 Chld.(17 from Japan, 18 from Serbia) ages 3 to 13 with autism

10s video fragment

5EA

6 engagement level [0-5] = evasive, non-compliance, indifferent, low engagement, mid engagement, high engagement

N/A

Ninaus et al. (2019)

ER

S

1) The number line estimation task, 2) watching a short clip.

122 participants

Image frames

Self-report

joy = “excited” or “inspired”, activity/interest = “attentive”, “active”, afraid = “distressed”, “scared”, upset = “irritable”, “hostile”

N/A

Yue et al. (2019)

ER

S &WE

MOOC course titled “Data Processing Using Python” with course 5=10 mins videos, teaching materials and quizzes.

46 participants

7224 learning performance instances

self-report and quiz score

7 emotions (Neutral, Happy, Disgust, Sad, Surprise, Fear, Anger),Eye Movement (writing, read, type), course: score

N/A

(AffectNet) Mollahosseini et al. (2019)

ER

W

Images collected from internet

450,000 subjects

Training set: 23,901 images Validation set: 3,500 images

12 EA for 450,000 images. 2EA for 36,000 images

’8 emotion categories (neutral, happy, ,sad, surprise, fear, disgust, anger, contempt), valence and arousal (continuous)

Yes

Celiktutan et al. (2019)

E

S

HHI: dyadic interactions, HRI: triadic interactions

18 students

290 clips of HHI, 456 clips of HRI, and 746 clips in total for each data modality (276 physiological clips of HHI)

self-report

Big Five personality traits (extroversion, neuroticsm, openness, agreeableness, conscienctiousness), 10-point likert scale of engagement

Yes\(^{f}\)

Youssef et al. (2019)

E

S

Interaction with Pepper robot

278 users (182 M, 96 F)

209 interactions featuring a single user, and 69 multiparty interactions

2 EA using ELAN

Sign of Engagement Decrease (SED), engaged

Yes\(^{g}\)

Olivetti et al. (2019)

E

S

A virtual learning environment (A European Enterpreneurship VET Model and Assessment)

12 participants (6 F, 6 M)

3D videos

2 EA and self-report

Engagement level 1,2,3

N/A

Ashwin et al. (2020b)

E

S &P

Offline classroom

50 students

24000 posed images of 50 students, 36000 images spontaneous

slef-anotate and 2 EA

Engaged, boredom and neutral

N/A

Ashwin et al. (2020a)

E

S &P

Offline classroom

350 students (Indian)

2900 posed images (1450 are multiple students in a single frame), 72000 spontaneous images

30 EA

Attentive (happiness, surprise, delight, engaged), in-attentive (sadness, fear, disgust, boredom, sleepy, frustrated, confused).

N/A

Pabba et al. (2022)

E

S

Offline classroom

50 (31 M and 19 F) (Indian)

1193 images (30 minutes)

5 EA

Engagement level academic affective states (low:Boredom, sleepy;Medium: Yawning, frustrated, confused;High: Focused)

N/A

Duchetto et al. (2020)

E

S

227 people (122 F, 105 M. 138 adults, 89 minors)

3,106 videos (10 fpr)

3 EA Using NOVA

Engagement score :High, low, medium

N/A

Yun et al. (2020)

E

S

Interactive multi-intelligence material

20 children (Asian)

356 video/images

3 and 7 EA

Engaged, Disengaged ([High Engagement, Low Engagement], [Low Disengagement, and High Disengagement]) [17:11:5:1]

N/A

Zhang et al. (2020)

E

47 students (28 M,19 F)

26 hours video (2 seconds image) and mouse movement

8 EA

1-5 engagement scale (but only 2 class classification Engaged, not engaged)

N/A

Liao et al. (2021)

Used Public Avilable dataset: Dataset for Affective States in E-Environments (DAiSEE)

Li et al. (2021)

E-R

S

A virtual patient in BioWorld

61 medicalstudents

167 segments, videos (10 seconds)

self-report, 1 EA

8 clinical behaviors, 2 performances (shallow/surface, high/deep)

N/A

Bhardwaj et al. (2021)

E

 S

Online class

1000 participants

Emotion scores

10 EA

0-5 scale engagement level and emotions: angry, disgust, fear, sad, surprise and neutral

Yes\(^{h}\)

Goldberg et al. (2021)

E

S

offline classroom (90 mins), knowledge test

52 students (only 30 were used due to occlusions)

Videos

self-report and 6 EA using CARMA

-2 to +2 engagement scale:

Chatterjee et al. (2021)

E

S

Dyadic conversation

16 dyads

Naturalistic conversations (15 minutes)

Self-report

Engagement level (none to very high), Engagement scale (0-100)

N/A

Youssef et al. (2021)

E

S

Interaction using Pepper robot

195 participants (70 F, 125 M)

124 interactions to feature a single user, 71 multiparty interactions (40 started as multiparty and ended as single user)

EA

Signs of User Engagement Breakdown (UEB): Breakdown, No Breakdown

N/A

Sumer et al. (2021)

E

S

Offline classroom

15 students

360 audio-visual recording

2 EA using CARMA every second

3 engagement class label (0,1,2)

N/A

Trindade et al. (2021)

ER

WE

Courses in Moodle

2752 Moodle record data from 2015-2019

N/A

Ma et al. (2021)

Used Public Avilable dataset: Dataset for Affective States in E-Environments (DAiSEE)

Thiruthvanathan et al. (2021)

Used Public Avilable dataset: DAiSEE, iSAFE, ISED

Altuwairqi 2021 et al. (2021b)

E

S

Writing task

42 participants

164 videos, mouse and keyboard log

Self-annotation

Strong, high, and medium engagements

N/A

Vanneste et al. (2021)

E

S

On lectures (hybrid virtual classroom)

14 students (4 F,10 M)

1031 clips (only 37-185 were annotated)

Self-report, EA

0,1,2 engagement

N/A

Hasnine et al. (2021)

E

S

Interactive lecture, lecture video taken from YouTube (28s)

11 students

N/A (concentration index (CI))

Highly-engaged, engaged, disengaged

Delgado et al. (2021)

E

WE

Math problem on MathSpring.org

19 students

400 videos(18,721 frames)

3 EA

Engaged (looking at the screen or looking at their paper), wandering

N/A

Engwall et al. (2022)

E

S

Robot interaction (with Furhat anthropomorphic robotic head) in Wizard-of-Oz setup

33 language learners

50 audio-visual conversational videos (38 video recordings, 353 of 5s clips)

1 EA (audio recordings), 3 EA (video recordings), 9 EA (2s clips)

High and Low engagement. Clips (very dissengaged, dissengaged, neutral, engaged, very engaged)

N/A

Mehta et al. (2022)

Used Public Avilable dataset: Dataset for Affective States in E-Environments (DAiSEE)

Dubovi et al. (2022)

ER

S

The Medication Administration Test (MAT), PANAS questionnaires

61 nursing students

Data streams, and pre-and post test context knowledge test

Self-report using PANAS

10 positive emotions and 10 negative emotions Positive and Negative Affect Scale (PANAS)(Watson et al. 1988)

N/A

Thomas et al. (2022)

Used Existed dataset\(^{i}\)

Shen et al. (2022)

Used Public Avilable dataset: JAFFE, CK+, RAF-DB

Apicella et al. (2022)

E

S

Cognitive task (Continuous Performance Test), background music, social feedback

21 students

45 seconds acquisition EEG signals

Self-report, Performance index

High or low emotion engagement, high or low cognitive engagement

N/A