RQ1: Can subtitles generated by an auto-subtitles system facilitate learning comprehension reach the same level as edited subtitles on an educational video?
The results of the quantitative analysis showed that students using the auto-subtitles systems had higher mean scores for learning comprehension on how well they could remember the content of videos than those in the no-subtitles group. Additionally, there was no statistically significant difference between the auto-subtitles systems group and the edited subtitles group in mean video content-learning comprehension scores. As a result, the effects of subtitles in learning, which influence learning comprehension, can be explained by the appearance of virtual-text information and translation, which are described further below.
Firstly, the effects of using an auto-subtitles system appearing on educational videos can be explained by Mayer’s cognitive theory of multimedia learning. That is because a video’s visual–textual information positively affects learning captioning and can facilitate learning by helping students understand the content, improve concentration, and support disabled learners (Mayer, 2014). Furthermore, while watching an animated video, students mentioned they could listen to the audio information and read the linguistic information displayed on the screen simultaneously. This finding is similar to previous studies that emphasised how video captioning allows learners to realise the content because it promotes language learning by facilitating students to visualise what they hear (Danan, 2004). Therefore, subtitles in educational videos influence students’ cognitive process in understanding the content and facilitating learning more than just listening to the audio.
Additionally, student number 60 commented on the use of subtitles in educational videos by suggesting, “Subtitles make viewers understand content easier and faster. The subtitles with keywords in videos can help with taking notes in learning.” This opinion is consistent with previous studies’ assumption that the videos’ keyword-focused subtitles will effectively enhance student learning (Cao et al., 2019; Hsieh, 2019). Thus, subtitles with keywords will help to emphasise the video’s message in their verbal language by allowing learners to grasp the concept from the videos.
Secondly, the reliance on translation is related to language proficiency. The outcomes revealed a positive relationship between language proficiency (listening comprehension) and learning comprehension of video content with and without subtitles. This correlation is consistent with student perceptions of language proficiency level, which refers to language proficiency as a barrier to learning through video content in a foreign language that students feel uncomfortable or unable to communicate. Foreign language proficiency problems affecting learning through videos are consistent with the previous study by Nonthamand (2017). Also, learners with high language skills are more likely to understand video content than those with low language skills (van der Zee et al., 2017). According to the learning satisfaction questionnaires, students in the group without subtitles stated that it was too difficult to listen to and understand the information since they were not good at English. While students in the edited-subtitles group believed that subtitles help non-native English speakers understand the video’s content. Therefore, subtitles are especially helpful for learners with low language skills or who need tools to understand a non-first language. This finding is consistent with the bilingual dual-coding theory (Paivio & Desrochers, 1980), which asserts that humans have a good memory when they receive information from more than two channels (i.e., verbal or auditory information and image or visual information).
Furthermore, subtitles in L1 might assist students in improving their learning of the L2. They commented on the benefits of utilising their L1 in their learning and practising English as their L2 in this study. From the open-ended questions, students reflected on watching videos with subtitles to learn others’ content in English. They also believe they can learn other subjects while practising English listening and learning new vocabulary. Therefore, when utilising subtitles in an educational video in English as an L2 course with native Thai-speaking learners, the students realise the benefits of using subtitles in their learning of foreign languages. These comments are consistent with Danan’s (2004) conclusion that subtitles play a critical role in translating verbal information in L1 into L2, increasing language comprehension, and leading to additional cognitive benefits, such as greater depth of processing. Added to this is the assumption of Guillory (1998) that reading subtitles are an easily performed act that makes learners feel comfortable. Additionally, the message translated from machine translation facilitates students’ vocabulary learning (Fuji, 1999; Hsieh, 2019; Lee, 2019; Niño, 2009, 2020). As a result, the Thai subtitle from the automatic subtitle system, which was conducted in this study, could improve participants’ comprehension of video content in English in computing science subjects.
RQ2: Do auto-subtitles and edited subtitles have different levels of cognitive load?
According to the results of this study, there was no significant difference among the three types of subtitles on levels of cognitive load. More specifically, this project’s results matched Chan et al.’s (2020) as they both found that auto-generated captions and correctly edited captions have no significant effect on cognitive load. The causes behind these results will be discussed in this section. Regarding the results, the four following issues could demonstrate why no significant difference was found between the experimental groups and the control group:
First, edited subtitles in Thai may be wordy and complex due to the translation process affecting the learner’s efforts to understand. A student in the edited subtitles group gave their opinions on the use of educational videos with subtitles as follows:
I want you to be careful about the translated subtitles because some parts of them can’t convey the concepts. There are too many translated words into Thai making it difficult to read. Maybe transliteration words are easier to communicate than translating into Thai. (Student number 41)
The student’s feedback demonstrates an attempt to comprehend content by matching the narrative in English with the Thai subtitles that appear. The cognitive load arising from the simultaneous encoding of information in two languages is consistent with the theory of bilingual dual coding by Paivio and Desrochers (1980). These connections between the L1 and L2 occur between translation-equivalent logogens. This interaction could be regarded as a subset of verbal associative connections with a high likelihood of activation in code-switching tasks (Paivio, 2014). Consequently, students may handle a cognitive load while encoding in bilingual and simultaneously interpreting differences in language structures when reading subtitles in Thai.
Second, the requirement for subtitles may vary depending on learners’ English proficiency. Since the participants had a high score on the English listening comprehension test (M = 10.15, SD = 3.49), learners in this study may have relied on their English language skills to interpret the video content rather than using Thai subtitles. This finding is reflected in the opinions of one participant who reported total English language proficiency scores and stated, “Listening to audio is easier than reading subtitles to understand content. Also, subtitles in English may be better than subtitles in Thai” (Student number 47). This opinion demonstrates that students with high English language skills may not rely on Thai subtitles when watching videos in English. This result is similar to Niño’s (2020) study, which found that students with higher language proficiency were more aware of the inaccuracy and claimed that they do not require the translation output from online machine translations for comprehension purposes. Moreover, this finding again relates to the bilingual dual coding theory (Paivio, 2014) because when students have a high level of bilingual proficiency, they may select whichever language option minimises the cognitive load that would be incurred by dealing with simultaneous bilingual inputs. In consequence, this could be assumed that the students in the experimental subtitling group may not have given the weight to understand the video content by reading the subtitles.
Third, students may suffer from cognitive overload and be unable to decode all information while experiencing cognitive overload due to watching videos containing too much verbal and visual information (Sweller et al., 1998). In this experiment, students using editorial subtitles reported that sometimes it is difficult to concentrate while reading subtitles and that low concentration is an obstacle to learning the video’s content. Nevertheless, there is a difference in the appearance of video subtitles on the YouTube platform, with automated subtitles being synchronous word-by-word, whereas editorial subtitles are whole sentences. Due to this difference in the formation of subtitles, learners may concentrate more on the gradually emerging stimuli and less on the content. This result is consistent with Shadiev et al. (2015)’s learning concentration study in an auto-caption system that allows learners to track the content being lectured and Kruger et al. (2017)’s study, which found that subtitles increase immersion with characters that fulfil a focusing effect to gain a stronger connection with the story.
Fourth, students may already have background knowledge of the topic being studied. At the end of the learning comprehension assessment, students can review their grades and the correct answers.
In one case, a student in the auto-subtitles group took the time to inquire about an incorrect test answer in the first video after the class ended. During the exchange, the student argued that his answer should be correct based on his prior knowledge of the topic. This discussion is noteworthy because the learner’s prior understanding of the additional content was not presented in the lecture video. This behaviour suggests that in experiments conducted, students may use prior knowledge to learn rather than learning new information contained in video learning activities.
As a consequence of the notions mentioned above, it was determined that there was no statistically significant difference in the level of cognitive load found between the different Thai subtitle formats (i.e., auto-subtitles, edited subtitles, and no subtitles) in this sample of computer science classrooms using educational videos in English.
RQ3: Are students satisfied with using auto-subtitles systems?
According to this study’s results, students in the auto-subtitling group were more satisfied with using Thai subtitles in educational videos in English. Providing subtitles in Thai as a first language makes students feel more comfortable because they can readily access meaning while watching foreign-language media (Guillory, 1998). In addition, the participants in the auto-subtitles group highlighted the advantages of utilising subtitles to learn through educational videos and that they are delighted to use the auto-subtitle system for watching others’ videos in foreign languages. They also mentioned that they could activate foreign language video clips they want to study for better understanding. These opinions demonstrate that the auto-subtitles system in the Thai language is beneficial to their learning, such as taking notes from the subtitles translated into Thai language or implementing the auto-subtitles system to translate other videos from foreign languages. Reading the subtitles while listening to and watching the video is also helpful in recapitulating the content (Liyanagunawardena, 2021). Therefore, using automated subtitles in educational videos in English is beneficial for improving positive learning attitudes among the sample participants.
This study found that the Thai subtitle generated by the auto-subtitles system in English educational videos could help students who require the translation subtitles to understand the video’s content in a language they are comfortable with. Although the quality of translated subtitles from the auto-subtitles system may be a concern for implementation in some languages (Niño, 2009; Briggs, 2018), these experiments with Thai subtitles generated from the auto-subtitles system in educational videos in English could be used in comparison with the study of using the same system of translation from English into Thai (Wongprom & Thitthongkam, 2018; Yuchareon, 2017). However, before using an automatic subtitle system in the classroom, instructors should ensure that the generated subtitles have been translated accurately.
This study had limitations related to the sample population. During the Covid-19 pandemic recruiting samples from schools ready for online learning and experimentation learning activities is a concern. In particular, it must include the requirements of schools and participants equipped with devices and network connections. As a result, the experiment of the sample group was limited to only one school in Thailand’s capital city, Bangkok. The sample group contained students with high background knowledge and advanced English skills due to social context. Moreover, the sample size was small, and the experimental period was limited. Plus, it was necessary to remove some students’ records who did not participate in both trials due to the students’ leaving the class for vaccination on the day of the experiment.
Furthermore, because this study experiment applies a Posttest-Only Control Group Design, it is difficult to discern the student’s prior knowledge of the video content, which could affect their learning comprehension and cognitive load. As a result, having a pre-study test is one method that could assist in controlling the experiment’s results more precisely.
Lastly, the instruments used in this study were adapted from previously reviewed studies on translating English to Thai instead of creating new instruments specifically in the Thai language. Also, due to the limited time and number of schools for this experiment, there was no opportunity to pilot the instruments before collecting data. Therefore, there may be restrictions on references to population groups due to the limitations mentioned above.
Suggestions for future study
From this study, there are related issues arising from observations during the experimental as follows:
Firstly, the videos used in this study differed in presentation. The first video in the experiment was a geometric motion graphic with no virtual graphics. On the other hand, the second video had redundancy animation, which is a visual representation of a machine’s algorithm that includes keyword text as cognitive cues. Interestingly, the results of the video learning comprehension scores following the first video revealed that students who watched with and without subtitles performed statistically differently. However, there was no statistically significant difference in the learning comprehension scores after the second video.
Secondly, this experiment should be replicated with a larger sample, hierarchically dispersed model with a wide variety of language ability levels for the results to be referenced to a population. Additionally, standard language proficiency levels, such as the Common European Framework of Reference for Languages (CEFR), may be classified in future studies to distinguish results based on the learner’s level of language proficiency. In addition, experiments should include a pre-study test of students’ prior knowledge related to learning content.
Lastly, at the time of this research experiment, there was no automatic subtitle system in the videos that could translate Thai into other languages. As a result, studying and developing an automated subtitle system that supports Thai translation in videos will greatly benefit open education broadly.
Consequently, for future research on the implementation of subtitle system tools in educational videos, the recommendations above should be considered to resolve any weaknesses and be used as caution in future studies.