Video annotation and analytics in CourseMapper

Video annotation

Yousef et al. (2014a) critically analyzed the current research of VBL in the last decade to build a deep understanding on what the educational benefits are and which effect VBL has on teaching and learning. The authors explored how to design effective VBL environments and noted that in addition to authoring tools for VBL content, such as lecture note synchronization and video content summarization, annotation tools are the most used design tools in the reviewed VBL literature. Video annotation refers to the additional notes added to the video, which help in searching, highlighting, analyzing, retrieving, and providing feedback, without modifying the resource itself (Khurana and Chandak 2013). It provides an easy way for discussion, reflection on the video content, and feedback (Yousef et al. 2015b). Several attempts have been made to explore the potential of video annotation methods to increase the interactivity in VBL environments for various purposes. In the following, we analyze the existing video annotations tools and summarize their applicability and limitations and point out the main differences to the video annotation tool in CourseMapper.

We selected seven video annotation systems for our analysis due to their potential of supporting collaboration in VBL environments. These include VideoAnnEx (Lin et al. 2003), the Video Interaction for Teaching and Learning (VITAL) (Preston et al. 2005), MuLVAT (Theodosiou et al. 2009), WaCTool (Motti et al. 2009), the media annotation tool (MAT) (Colasante 2011a), the Collaborative Annotation Tool (CATool) Open Sourcing Harvard University’s Collaborative Annotation Tool 2016, and the Collaborative Lecture Annotation tool (CLAS) (Risko et al. 2013).

The analysis shows that all tools support basic features of video annotation, such as providing annotation panel, video controls, viewing area, custom annotation markers, and external discussion tools e.g. wiki, blog, chat. Only CATool and CLAS are providing more advanced features, such as social bookmarking and collaborative discussion panels. Additionally, the lack of integration between these tools and learning management systems or MOOCs makes their usage unpractical and out of context.

As compared to these tools, CourseMapper uses a relatively new approach of representing and structuring video materials where videos are collaboratively annotated in a mind-map view. CourseMapper provides the opportunity to better organize the course content by different subjects. Moreover, annotations are updated in real-time and can be embedded inside the video. The social bookmarking, discussion threads, rating system, search engine, as well as filtering and ordering mechanisms for annotations were built into CourseMapper to support a more effective self-organized and networked VBL experience.

Video analytics

Despite the wide agreement that learning analytics (LA) can provide value in VBL, the application of LA on VBL is rather limited until now. Most of the LA studies have been done in a MOOC context and have focused on an administrative level to meet the needs of the course providers. These studies have primarily focused on addressing low completion rates, investigating learning patterns, and supporting intervention (Chatti et al. 2014). Further, only little research has been carried out to investigate the effectiveness of using LA on activities around video content.

We begin our review by looking over the "What?" dimension of the reference model and also take a look at the experiment setting and the tool lifecycle. With the vast development of analytical tools, the standard research activities have been conducted as a controlled experiment. This is still a popular environment, where tools can be modified with such requirements, so that "noisy" results can be avoided and focus can be targeted towards specific features. Several studies used namely this experiment setting (Brooks et al. 2011; Colasante 2011b; Giannakos et al. 2015). In general, the gathered data usually comes from in-house frameworks and applications or surveys and observations conducted within the institution. And, most of the tools are not developed for reusability in third-party environments.

The video learning analytics system (VLAS) is a video analytics application designed for use in a video-assisted course (Giannakos et al. 2015). The authors have used the trace data generated by students interacting with VLAS, including their video navigation history and combined the results with student learning performance and scores gathered from system questionnaires. The system has a reusable lifecycle and it is constructed with open-access to the general public.

Pardo et al. (2015) and Gasevic et al. (2014) used data collected from traces of CLAS. CLAS is a Web-based system for annotating video content that also includes a learning analytics component to support self-regulated learning (Mirriahi and Dawson 2013). Both experiments were conducted in a natural environment. However, the first study used trace data collected from MSLQ tool, midterm scores, number of annotations and covariates derived from MSLQ and SPQ questionnaires as additional data sources. In contrast, the second research included assignment of participants to two different experimental conditions, annotation counts, and LIWC special variables for linguistic analysis.

The study in (Brooks et al. 2011) was also conducted in a controlled environment. The authors used the "Recollect" tool event monitor trace data, interactions of users with player, events collected from player’s "heartbeat" mechanism, student questionnaires as an input source. Guo et al. (2014) provided a retrospective study that used edX trace data, interviews with edX staff, page navigation, video interactions and submitting a problem for grading as sources of data.

CourseMapper uses traces collected from students’ interaction around the video content (What?). The LA component of CourseMapper was designed with the general idea of reuse. Therefore, it is not limited to the research environment and can be applied in both natural or controlled experiments. To note that in a long-term usage of CourseMapper, the collected data within its database can be used to support retrospective studies.

Next we examine the “Why?”, “How?” and “Who?” dimensions of the LA reference model. We noted that most of the studies had researchers as the main target group. Only the study in (Colasante 2011b) addressed teachers and learners as primary stakeholders. Further, most of the studies used machine learning and data mining techniques for different purposes and statistics to present the analytics results. Brooks et al. (2011) used k-means clustering to help researchers investigate students’ engagement with video recorded lectures. The methodology clustered students based on video tool access. The main objectives in this work were to support monitoring and analysis, show that analytics in learning systems can be used to provide both auditing and interventions in student learning. Data mining was also applied in (Guo et al. 2014) to see how video production decisions can affect students’ engagement. The goal of the study was to give recommendations to instructors and video producers on how to take better advantage of online video formats. Linear regression was used in (Pardo et al. 2015) to investigate the impact of video annotation usage on learning performance. And, Gasevic et al. (2014) used statistical analysis to explore the usage of video annotation tools within graded and non-graded instructional approaches.

Only two studies used information visualization methods based on simple charts, namely (Giannakos et al. 2015) to investigate relationships between interactions with video lectures, attitudes, and learning performance and (Colasante 2011b) to investigate the effectiveness of the integration of the video annotation tool MAT into a learning environment.

CourseMapper aims at fostering effective personalized learning and supporting both learners and teachers (Who?) in monitoring, awareness, self-reflection, motivation, and feedback processes in a networked VBL environment (Why?). It uses traces collected from learners’ interactions to build heatmaps reflecting the most viewed parts of the video. Moreover, it uses the start/end time of annotations to produce annotation maps that stacks and highlights the frequently annotated areas of the video (How?).

Technologies

In the server side backbone of CourseMapper lays Node.JS and Express Framework. Node.JS provides great event-driven, non-blocking I/O mode, which enables fast and scalable applications to be written in plain JavaScript (JS). Node.JS has a very steep learning curve and its default callback based programming style makes it harder for developers to write any blocking code. Express is a minimal and flexible Node.js web application framework that provides a robust set of features for web and mobile applications.

In order to provide real-time annotation updates and editing, CourseMapper has integrated Socket.IO engine. It bases the communication over WebSockets, however it does not assumes that they are enabled and will work by default. At first it establishes a connection with XHR or JSONP and then attempts to upgrade the connection. This means that users with browser, which does not support WebSocket-based connections will not have any degraded experience. Persistent login sessions are established via Passport.JS middleware, supporting multiple authentication schemas, including OAuth. Upon their choice users can select to login with their Facebook account and do not maintain one within the system.

Application data is stored inside MongoDB a cross-platform NoSQL document-oriented database. It substitutes the traditional table-based relational structure with JSON-like documents, which allows data easier and faster data integration. In order to simplify client side development and testing CourseMapper uses Angular, a framework providing modelviewcontroller (MVC) and modelviewviewmodel (MVVM) architectures, along with commonly used components.

For content playback, CourseMapper uses Videogular. It is an HTML5 video player for AngularJS. The player comes with default controls and multiple plugins, such as several scrub-bars, cue points (a way to trigger functions related to time) and many more. Videogular also significantly simplifies the way new plugins and controls can be developed, styled and integrated into it.

Components

The video annotation section workspace of CourseMapper can be seen in Fig. 1. It consists of a video player and several components that are listed below. A general note to take is that there are many other features of CourseMapper, which we will not describe in this paper in order to focus mainly on the video annotation and analytics parts of the platform.

Annotation viewer

The annotation viewer is a system component that loads existing annotation from the server via WebSockets and reflects any changes in real-time. Each annotation is displayed in its own container and further comments can be made when the comment section is expanded, as shown in Fig. 2.

Annotation editor

The CourseMapper annotation editor allows users to create or update existing annotations. It is a user control placed within the layout of the annotation viewer and hosts editors for each field of the annotation model, such as text, start time, end time, annotation type. It is important to note that everyone can create annotation, however only moderators which are listed for the current course or annotation owners can edit and update the content of an existing annotation. A snapshot of the control can be seen in Fig. 3.

Embedded note vs note

CourseMapper enables users to distinguish between two different types of annotations, namely notes and embedded notes. However, they can be mutually exchanged for a single annotation, or to be more precise an embedded note can be easily converted to a note or vice-versa.

Note

is an annotations that is bound to a specific timeframe within the video content, however it is only displayed inside the main annotation viewer control. A note inside the annotation viewer is activated and highlighted when the current player position crosses and stays in between the start/end time of the annotation. Once the player position exits this window the annotation is therefore marked as completed, it gets deactivated and visually grayed out in order to avoid disturbing the viewer’s attention further on. As an addition this behavior can be seen as two-way binding, due to the fact that if an annotation from the annotation viewer is clicked, it will transition the video player to the start time of the annotation, allowing easy navigation between important parts of the media.

Embedded note

is an annotation that possesses all features of a regular note with an addition of pointing a specific "hotzone" - an opaque rectangular which is overlaid on top of the video content. The rectangular zone’s position and size can be edited and stored as a supplement to the annotation model. Both dimensions are relative and restricted to the maximum of those of the video player’s container. This way a user can specify an important part of the content and focus views attention to it. Whenever the embedded zone gets hovered over inside the player it will display the annotation’s text (see Fig. 4). This features is of a significant use in full screen mode, when the annotations viewer and the rest of the application is not visible.

AnnotationMap scrub bar

AnnotationMap is a visual learning analytics component of CourseMapper that extends the regular scrub bar, as shown in Fig. 5. It overlays stacks of annotation windows within the given timeline. It is placed in the controls panel of the video player. In order to keep the user confusion minimal and simplify the visual seeking for annotations the cue points here are displayed in opaque yellow color. The stack zones of overlapping annotation times will sharpen and brighten in a yellow nuance, notifying the viewer that this portion of the video timeline has a larger congregation of annotations and most likely contains interesting information.

Heatmap scrub bar

Heatmap is another visual learning analytics component of CourseMapper. Whenever a student navigates back and forward and interacts with the player he leaves his "footprint", which contributes to the overall heatmap. The Heatmap control extends normal scrub bar with a heatmap based color scheme, where the most viewed parts of video are marked with warm colors such as orange and red, neutral are shades of the yellow spectrum and less viewed parts are usually displayed with cold purple and blue colors, as depicted in Fig. 6. Based on this picture students can visually scan and easily find the most interesting areas of the video. Moreover, the Heatmap shows how many times the video has been watched.

The Heatmap module consists of five parts, two on a server side and three on a client side. The server side provides common API for all clients. All received data is processed and stored on the server side, NodeJS and MongoDB work together in order to process requests as fast as possible and to support large numbers of users online. The server side provides two routes:

• GET/get - returns data of the particular page based on request headers. It is not possible to specify page URL, this decision will be made in automatic manner on the server side.

• POST/save - saves or update data of the particular page based on request headers.

The main task of the client side is to avoid all interaction with the structure of a host system or web site. It consists of three parts: Observer, Heatmap and Viewer. Each part has its own task, for instance, the Observer has to handle all important events in order to track user behavior. It also handles special types of events about a state of a user, like "idle" or "active". The Heatmap uses HTML5 canvas in order to represent input data using predefined colors. And finally the Viewer is a part which mostly interacts with the host system. It fetches data and embeds heatmap in content viewer. In the next sections, we discuss the implementation of these parts in more details.

Observer

The Observer class is used to collect information about how users view a content and then send the data to the server side using POST/save AJAX call. HTML5 Video provides API to get such events like play, pause, stop, seeking, etc. The Observer class subscribes to those events and listens for all actions that user makes while watching a video. Each time when a user is watching some part of a video the Observer stores start point as a value from 0 to 1. For example, if a user starts watching from the middle of a video the Observer will save new start point - 0.5. In the same way Observer stores endpoint of a watched video.

Heatmap

The Heatmap component is a basic implementation of 2D heatmaps called "simpleheat". However, instead of 2D, FootPrint implementation works in 1D space. As an input, LinearHeatmap accepts an array of values and maximum possible value. LinearHeatmap is a light implementation of linear heatmap that allows precise heatmaps configurations. The colorization algorithm works as follows:

1. 1.

   At first LinearHeatmap generates color palette which will be used to set correct colors in draw function. This step passes only once.

    
2. 2.

   LinearHeatmap builds grayscale gradient using standard canvas API. The result of the first step will be black linear gradient with different values of alpha.

    
3. 3.

   Based on alpha value in each pixel LinearHeatmap applies correct color that is stored in color palette

    

Viewer

The main task of the Viewer class is to extend regular controls with generated heatmap. Video Viewer uses standard HTML5 player and adds an additional slider on the top part of a video. This slider based on custom HTML and CSS with canvas element inside, that is used by LinearHeatmap class to draw a heatmap. Additional slider shows "hottest" or most viewed parts of the video. At the same time, Observer class gathers data about viewed parts of the current user and each viewing of some part of a video is a contribution to the entire heatmap.

Scenario

We used CourseMapper in the eLearning course offered at RWTH Aachen University in summer semester 2015. We conducted a controlled experiment to evaluate the Heatmap module in supporting an effective networked and personalized VBL experience through the support of awareness, reflection, motivation, and feedback. We evaluated the Heatmap module as part of an exam preparation scenario. The beginning of the semester is quite flexible, because this is time for overview of lectures and first assignments. Throughout the semester, the workload is increasing and approximately 2–3 weeks before examination, students have to go through significant amount of learning materials. In the evaluation, we simulated a real exam preparation setting. The students were provided with a list of possible exam questions from the last years. They were asked to use the provided video lectures to get answers to the questions. The students were then split into two groups. The first group had to go through the video content without Heatmap.

We then conducted an evaluation of the Heatmap module in terms of usability and effectiveness. We employed an evaluation approach based on the System Usability Scale (SUS) as a general usability evaluation and a custom effectiveness questionnaire to measure whether the goals of monitoring, awareness, reflection, motivation, feedback have been achieved through the support of the Heatmap module. The questionnaire also includes questions related to user background, usage of learning materials, and user expectation of analytics on learning materials. Ten computer science students and three teachers completed the questionnaire.

User background

The first part of the questionnaire captured the participants’ backgrounds. Figure 7, shows that most students very often use online materials. The most popular materials are slides and students are able to find very quickly the right information using regular search commands. The second most popular online material are video lectures. However, the survey shows that students experience some difficulties searching for information within video content. Finding important information in a video is a hard task especially if the student has not attended the lecture. The video has no titles, images, and paragraphs, the only way to search is to rewind and keep watching. Also, students admitted that they use printed books rarely. In general, the survey results confirm that learning is increasingly happening through digital resources and that videos represent an important medium in today’s learning environments.

User expectation

The second part of the questionnaire captured the expectations on the features that the users generally would like to have in an analytics tool on learning materials. The user expectation evaluation showed that most of the students want to quickly locate important parts of learning materials and to understand how other students use them. They pointed out that improvements in this direction would make the learning process more efficient and effective. On the other hand, teachers are interested in getting information on which learning materials are used more frequently and how they are used.

Usability

The third part of the questionnaire dealt with the usability of the tool based on the System Usability Scale (SUS) which is a simple, ten-item attitude Likert scale giving a global view of subjective assessments of usability (Brooke 1996). The questions are designed to capture the intuitiveness, simplicity, feedback, responsiveness, efficiency of the tool, and the steepness of the learning curve which a user must go through to successfully use the tool. Figure 8 shows the results of the usability evaluation using the SUS framework. The usability scale of the system is approximately 90, which reflects a high user satisfaction with the usability of the Heatmap module. In general, the respondents found the tool intuitive, easy to use, and easy to learn.

Usefulness

The fourth part of the questionnaire captured the usefulness of the tool. The usefulness evaluation consists of two parts, the first part is a questionnaire for students. This part covers questions related to dealing with information overload, monitoring, awareness, and motivation. The second part was created to evaluate the system from a teacher’s perspective and whether the Heatmap module can be used as an effective monitoring, reflection, and feedback tool.

Student perspective

Students of the first group did not use the Heatmap module while trying to answer the given exam questions. However, after the exam preparation task, we showed them their activities on the heatmap. Students of the second group used the heatmap right from the beginning. We asked students from the two groups to give their opinion on the Heatmap module as a potential LA tool that can support personalized learning in a VBL environment. As shown in Fig. 9, the majority of the respondents agreed that the tool can make the learning process more efficient and effective and that the tool has the potential to increase motivation through the monitoring of peer’s activities. Further, the respondents liked the feature that the Heatmap also provides information on how often a video has been watched, which can help them find popular videos, thus ovecoming a potential information overload problem. All respondents from the second group opined that the Heatmap helped them to find important parts of the learning materials. However, not all respondents were sure that they understood how other students use the learning materials. To note that respondents from the second group rated the capabilities of the Heatmap higher.

Teacher perspective

Figure 10 shows the result of the usefulness evaluation from a teacher’s perspective. The task for the teachers was to have a look at the results of the two student groups and to gauge whether the Heatmap can support monitoring, feedback, and reflection. The teachers agreed that the tool can help them monitor students’ activities and give a good feedback on the important/critical parts of learning materials. But not all teachers were sure that the tool can help with reflection on the quality of learning materials. The teachers, however, noted that this is due to the evaluation setting (i.e. simulation of an exam preparation phase based on predefined questions). They pointed out that the Heatmap can indeed be a powerful reflection tool if it was used throughout the whole semester.