Scaffolding support through integration of learning material

Abstract

The ever-growing amount of learning material poses a challenge to both learners and teachers. In order to perform a learning task, students often need to search for additional information. Inquiry might be hampered if they do not know what exactly to search for, especially if they lack prior knowledge in a domain. Furthermore, they might need to recall meta-information, e.g., the implicit links between learning resources or the location of an explanation within a document. In addition, their performance in a learning task might suffer from the constant need for further information. As the authoring of learning material is a time-intensive task, reusability of resources in different contexts should be enabled. In this paper, we present the ALMA-Yactul ecosystem, a holistic approach for student-centered integration of learning material. Based on semantic annotations, learners are provided with scaffolding support for retrieving material related to their current study task. This integration of resources is showcased in a variety of applications and domains, such as a plugin for an IDE or an enhanced sketchnoting app. Moreover, we show how Yactul, a gamified student response platform, can benefit from semantic annotations and integration of learning material. Additionally, teachers are provided with semi-automatic annotation support in the popular Office 365 suite to foster the reusability of their content. Apart from presenting the results of its usage in university classes, we also show how high school students without prior knowledge benefitted from this scaffolding support. A significantly better performance in resolving programming-related tasks could be observed compared to a traditional search process. Finally, the ecosystem is evaluated against a state-of-the-art technology integration model.

Introduction

Organizing a vast collection of learning material is a challenging, time-intensive task for both learners and teachers. Traditionally, while attending lectures, students follow presentations and take notes on the presented content. However, terms or concepts might be mentioned that are unknown to an individual student, yet not further specified by the teacher. For instance, in a history class about the Second Punic War, the battles of Cannae and Lake Trasimene might be mentioned, without indicating where these places are situated. Similarly, in a physiology course, a chemical compound could be referred to, which the student wants to know the structure of. To avoid losing track of the lecture, the student can hardly look this information up while in class. At home, she might struggle with going through her unorganized notes and getting back into the context of the lecture. Finding a certain note if the location or exact phrasing is unknown is an issue.

A practical task, e.g., an exercise, could also bring up a concept that requires further studying before being able to solve the task. Students would need to go through a potentially vast amount of learning material provided by their teacher. This requires to identify concepts in the material and memorize the association between a concept and a particular resource. However, complex ideas can be introduced at a fine-grained level. Additionally, implicit links between documents across different courses would also need to be recalled by learners. For instance, in a programming exercise, students could be asked to work with inodes, a concept that might have been introduced during an operating systems course in a previous semester. If the necessary information was not given in the resources provided by the teacher, students are likely to search the Web and use the resulting documents as learning material (Krieger 2015). Now, they would be confronted with an open corpus of heterogeneous resources of diverging quality. High quality learning material might go undiscovered (Sabourin et al. 2015). Students might feel lost in hyperspace (Edwards and Hardman 1999) and perceive a higher mental effort, as they would need to integrate information from multiple places (Hundhausen et al. 2017). In addition, the assumption that students know how to perform a search effectively is often not fulfilled, as they might lack prior experience with advanced search functionalities (Mavridis et al. 2017; Paynter 2015). The level of expertise and the search accuracy are often positively correlated (Dorn et al. 2013). As there is usually no explicit link between a task and the relevant learning material, it is especially challenging for novices to know what to search for, as already expressed in Meno’s paradox of inquiry. In addition, the continuous need to switch between a task and the search for related resources can lead to distraction or even abandonment of the task (Krieger 2015).

Furthermore, the experience in the classroom can be fostered by using active learning strategies, such as game elements. Gamification, i.e., the "use of game design elements in non-game contexts" (Deterding et al. 2011), can be provided through student response systems, such as Kahoot!, Socrative or Quizizz. However, existing gamification platforms often provide a limited set of activity types, do not keep track of an individual learner’s progress on a certain topic and do not exhibit a link between a game activity and the related learning material.

From the perspective of teachers, authoring high quality learning material is a time-intensive task as well. Therefore, the reusability of resources in different contexts should be enabled (Chukwunonso and Ikani 2015). For instance, a document on inodes from a first-semester course on operating systems should be easily retrievable in the context of a task in an advanced programming course. Using Semantic Web technologies, such as ontologies, in the metadata of learning resources may foster their interoperability, discovery, reusability, interlinking and integration into different contexts. However, authors of learning material, such as teachers or instructional designers, should neither require substantial technical knowledge, nor see a significant increase of their authoring time in order to semantically enhance their documents. Also, they should be able to do so in their usual authoring tools.

In this paper, we present the ALMA-Yactul ecosystem, a holistic approach for student-centered integration of learning material. Relying on semantic annotations, learning material from a dynamically expansible, heterogeneous set can be interrelated and integrated from different sources in different contexts, in a fine-grained and cross-curricular way. More precisely, learners will be able to pinpoint relevant resources for their current learning task. Opposed to a material-centered environment, students do not need to recall which resources cover a concept. Instead, they are provided with ad-hoc scaffolding support by integrating learning material into the environment of their learning task, thereby closing the physical gap between the application of a concept and its explanation. For the domain of computer programming, a plugin was developed that provides a student with learning material within Eclipse, a popular Integrated Development Environment (IDE). Furthermore, an enhanced tablet-based sketchnoting app has been created to help students retrieve related information on their notes without explicitly querying for it. Additionally, we present how Yactul, a gamified student response framework enabling a continuous active learning experience, keeps track of a learner’s performance on a per-concept basis and relates activities to matching learning resources. Additionally, teachers are given annotation support in popular authoring tools. A semi-automatic concept recognition process allows to keep the impact of annotations low, while granting reusability of their authored resources in the different contexts mentioned above. While our main use case has been computer science, it will be shown that the approach can also be applied in other domains. An experiment with high school students without prior knowledge in the domain of computer programming yielded promising results concerning the usability and effectiveness of our approach. Finally, the ecosystem is evaluated against a state-of-the-art technology integration model.

Background & related work

Psychological foundations

Information can come from different channels (Farhoomand and Drury 2002) and in heterogeneous nature. The field of Personal Information Management (PIM) aims at reducing this fragmentation such that people can retrieve information relevant for their current context and thus allow them to focus on the task at hand (Jones 2007). There are several issues to tackle. First, on a certain topic, learners could potentially have a vast set of resources, both locally stored or available on the Internet. If the time required for information processing exceeds an individual’s time capacity, this can lead to an information overload (Schick et al. 1990). Furthermore, in absence of an explicit link between a task and related documents through, e.g., cross-references or hyperlinks, they would need to remember implicit links, which is hardly feasible for a large set of resources. Beaudoin (2013) calls this phenomenon the meta-access problem, which "prevents people from fully capitalizing on knowledge resources".

According to Cognitive Load Theory (CLT), cognitive load can be divided into intrinsic CL (inherent to a subject matter), extraneous CL (caused by the instructional design) and germane CL (required for constructing schemas in the long-term memory) (Sweller et al. 1998). If a learner has to constantly leave the context of a task to search for related information, a so-called split-attention effect can be caused (Chandler and Sweller 1992), leading to an increase of extraneous cognitive load (Schmeck et al. 2015). It is widely accepted that our working memory is limited. Hence, an increase in extraneous cognitive load will affect the learning process, as germane cognitive load will be reduced to avoid a cognitive overload. This is especially critical if the subject matter is complex, causing a high intrinsic cognitive load. While the long-term memory is considered unlimited (Sweller et al. 1998), retention of information has to be fostered, e.g., through spaced repetition. Based on the work of Ebbinghaus (1885), active recall of information in specific intervals strengthens its retention. Well-known applications of spaced repetition are the Leitner system (Leitner 1972) known from flashcard-based learning, as well as the SuperMemo family of algorithms (Wozniak and Gorzelanczyk 1994).

Active learning can be defined as "any instructional method that engages students in the learning process" (Prince 2004). Active learners benefit from experimentation in order to gain new knowledge. However, learning material, often realized as static presentation of knowledge, lacks support for active learners (Kirsch et al. 2014). Active learning finds its roots in constructivism, a school of thought which assumes that learners can construct knowledge from experiences. While there exist unguided constructivist approaches, scaffolding is considered the intervention of a tutor to enable a learner to solve a problem beyond his unassisted efforts (Wood et al. 1976). It has been shown that scaffolding is superior to unassisted discovery learning, especially for learners without prior knowledge in a field or topic (Freund 1990; Klahr and Nigam 2004; Alfieri et al. 2011).

Organization of learning material

Current PIM tools fail to adequately address issues such as the meta-access problem (Beaudoin 2013), and would actually benefit from more information instead of less, if given at the meta-level of documents (Farhoomand and Drury 2002). Learning Objects are a common way of organizing learning resources, usually comprising content items, practice items and assessment items on a certain learning objective (Cisco Systems 1999). There exist different standards and schemas for learning object metadata, such as SCORM or IEEE LOM, leading to a fragmented landscape (Dietze et al. 2012). Learning Object Repositories (LOR), i.e., Web-based libraries providing learning objects, often rely on the metadata of learning objects for organization and retrieval purposes. However, there has been some concern about the completeness of such metadata as well as the capacity of current metadata schemata to effectively recommend LOs (Estivill-Castro et al. 2016; Palavitsinis et al. 2014; Krieger 2015). According to the statistics yielded by the Moodle platform used at our institution, SCORM packages are not used. Similarly, Open Educational Resources (OER) risk to remain underutilized as diverging knowledge schemes are used to tag them, which hampers their discovery and interoperability (Chicaiza et al. 2014; Navarrete and Luján-Mora 2015). Adaptive Educational Hypermedia Systems (AEHS), while excelling at adaptive recommendation and presentation of resources, often lack proper authoring tools (Sosnovsky and Brusilovsky 2015) or require teachers to have substantial technical knowledge (Watson et al. 2010). Their limited interoperability with other systems, such as Learning Management Systems (LMS), has also hampered their adoption by practitioners (Somyürek 2015).

Platform overview

The main goal of the ALMA-Yactul ecosystem is to provide a holistic approach for learning material integration and organization, for both learners and teachers. Figure 1 shows an overview of the platform. The ecosystem comprises tools for learning support (consumers of learning material), authoring support (producers of resources) and uses different distribution channels. In order to enable an unambiguous identification of concepts, Semantic Web technologies such as ontologies are used to annotate learning resources at the level of their metadata. With help of the producer tools, teachers add these semantic annotations to the learning material, publish them on the distribution channels, which are queried by the consumer tools to provide the scaffolding support to learners. The approach is holistic, as both teachers and learners benefit from the comprised tools, from the production of learning material to its consumption.

Fig. 1

Overview of the ALMA-Yactul Ecosystem

Among the consumers of learning material are a plugin for the Eclipse IDE, the Yactul mobile app, an enhanced sketchnoting app and SoLeMiO, an Add-in for the Office 365 suite. Both Eclipse and Yactul can be considered as active learning tools. SoLeMiO and Yactul are also used as producers of learning material, as will be explained in subsequent sections. Finally, as distribution channels, we created the ALMA repository, on which resources are published by the producer tools and are retrieved by the consumer tools. Furthermore, the Moodle LMS has been extended to take advantage of the semantic annotations within uploaded resources for its built-in tagging feature. Of course, the ecosystem is extensible for further producer and consumer tools as well as distribution channels.

Semantic foundations

For our main use case, namely computer science, we use the ALMA Ontology, an extensible and lightweight modular ontology for programming education (Grévisse et al. 2017a). Using an SKOS-based architecture, it contains modules comprising concepts from the Java and C programming languages, as well as operating systems. Each module realizes a fine-grained knowledge representation of its respective scope. By aligning concepts from different modules, learning material from different subjects can be brought into relation. The ontology is extensible with respect to further programming languages. Using a lightweight ontology is considered sufficient for annotation purposes (Andrews et al. 2012). Apart from this domain-specific ontology, we also make use of the domain-general DBpedia, a central knowledge base in the Linked Open Data cloud. This is especially used in the sketchnoting app, as well as in SoLeMiO, where semantic annotator tools, such as DBpedia Spotlight (Mendes et al. 2011) or TAGME (Ferragina and Scaiella 2010), are used for semi-automatic annotation purposes. The ALMA Ontology itself is also aligned with DBpedia, to grant interoperability outside the ecosystem.

Integration of learning material

According to the previous discussion, searching for high-quality learning material should be scaffolded, while including active learning resources and enabling a context-based, personalized learning experience. This should not be regarded as "spoon-feeding", rather than as transmitting good search practices and initial, high quality search results. By doing so, learners will become acquainted with the vocabulary used in the retrieved resources, fostering future inquiries and - hopefully - avoiding quick-and-dirty approaches, such as copying from the first result on Google. The workload is transferred from the search process to the actual task, fostering the creation of schemas. The integration of learning material is student-centered, as students do not need to proactively search for resources, rather than benefit from their provision in different contexts.

IDE plugin & hashtags

Apart from actively writing new programs from scratch, novice programmers benefit from reading small programs to establish pedagogical programming patterns (Nunes de Barros et al. 2005). However, if a part of the provided code is unclear, it is difficult for them to know what to search for. Additionally, general purpose search engines may not perform well for code queries (Dorn et al. 2013). For instance, if a student in a C programming course sees a line like int * p = &i;, he might wonder what * or & mean, but these special characters might be ignored by such a search engine.

A plugin was developed for the popular Eclipse IDE (Grévisse et al. 2017c). Following a similar approach than in Hosseini and Brusilovsky (2013), the plugin benefits from the Abstract Syntax Tree (AST), a tree representation of the structural elements of a piece of code. When a user selects a piece of code, the corresponding AST node is determined. Through a mapping between the node names specific to the language model of the IDE and the concepts of the ALMA Ontology, an ontological metamodeling (Pfeiffer and Wąsowski 2015) of the respective piece of code is established. Assuming that learning material has been annotated with the same concepts, linking a piece of code to related documents becomes feasible. The plugin currently supports both Java and C code. As shown in Fig. 2, the user selected the for-loop, for which learning resources of different types (websites, videos, PDFs) are retrieved from the ALMA repository and provided in the browser window at the bottom of the screen. The user can thus consult them from inside Eclipse, without leaving his current study context, namely the piece of code. Through the depth of the AST and the rich set of concepts in the ALMA Ontology, a fine-grained retrieval and integration of related learning material is possible, fostering the divide-and-conquer approach of understanding complex syntax. The AST view on the right-hand side shows the human-readable label of the concept from the ALMA Ontology. For instance, selecting a * in a piece of C code will show the label "Pointer", which leverages future queries of a learner by knowing the associated concept name. This approach could also be used in other IDEs or code editors, and further programming languages, as long as a mapping between AST elements and concepts in the ALMA Ontology is given.

Fig. 2

ALMA Plugin for the Eclipse IDE

For higher-level concepts not related to syntax, e.g., design patterns, or concepts not related to programming at all, we have been using hashtags on our lab sheets. Being ubiquitous on social media platforms such as Twitter, Facebook and Instagram, the use of hashtags in education should further engage students. Figure 3 shows an exercise from a second-semester course on C programming, which asks students to retrieve information from inodes. While the exercise actually focusses on the use of structs in C, it also requires to remember the concept of inodes, which was introduced in a first-semester introductory course on operating systems. The two hashtags next to the title represent concepts from the ALMA Ontology. Clicking on these hashtags opens the ALMA repository with the learning material for the corresponding concept.

Fig. 3

Hashtags on a lab sheet linked to the ALMA repository

Enhanced sketchnoting app

A recent addition to our ecosystem is an enhanced tablet-based sketchnoting app (Atashpendar et al. 2019). Sketchnoting is a neologism for note-taking and sketches, and has become increasingly popular in recent years (Dimeo 2016). Paper-based notes are difficult to search for, compared to their digital counterpart. However, current note-taking apps do not provide additional information on the content of notes. Our enhanced sketchnoting app assists in the above mentioned scenarios by providing scaffolding support to students while they are taking notes in the classroom. In a first step, the iPad app tries to recognize the handwritten notes using Google’s Firebase ML Kit. Then, the recognized text is passed through concept recognition and entity linking tools, such as DBpedia Spotlight or TAGME, in order to identify named entities in the notes. The resulting set of concepts can then be used to propose related resources from a knowledge graph, such as the abstract from the corresponding Wikipedia article, the structural formula of a chemical compound, or the map of a place, as shown in Fig. 4. The semantic information is stored in the metadata of a note, such that it can be sent along the note to another device. Apart from providing related information, the app also tries to recognize drawings. For instance, if a user previously drew a warning triangle next to a note to reflect that this part should be reviewed before an exam, he can later on filter his notes by redrawing the same shape. Finally, the app also allows to search for information that was not explicitly taken note of, but is mentioned in the related information, e.g., in the abstract of the Wikipedia article corresponding to a recognized named entity. For example, if a student remembers that the teacher mentioned a composer of the classical era, he could search for "classical" and be returned the note on Mozart.

Fig. 4

Additional information retrieved for a handwritten note in our enhanced sketchnoting app

Distribution channels

The ALMA repository, shown in Fig. 2, is used by different consumer tools, such as the Eclipse plugin or the Yactul app, as a “one-stop shop” (Shi et al. 2015) to retrieve heterogeneous learning material. Upon indexation of a resource, which can be stored on any server, the metadata is searched for concepts from the ALMA Ontology. The retrieval of material is contextual, as the information concerning the origin of the inquiry is sent altogether with the concepts for which a user wants to retrieve resources. For instance, the selected piece of code in Eclipse or the ID of a quiz activity in the Yactul app are sent, together with session IDs to distinguish users while assuring their anonymity. The collected data can be used for both learning analytics and adaptive recommendations of learning resources.

Apart from the ALMA repository, our ecosystem also provides an extension to the Moodle LMS (Grévisse et al. 2017b). In fact, learners can experience difficulties in finding a resource related to a certain topic if the number of resources in a Moodle course is high (Mazza et al. 2011). This is even aggravated when files lack a meaningful name (e.g., Lesson 1, Lab 5). Learners would need to sequentially go through all resources to find the necessary one. Starting version 3.1, Moodle includes a tagging feature, where teachers can manually add tags to resources. A tag block has also been introduced. Similar to the ALMA repository, our Moodle extension extracts concepts from the ALMA Ontology out of the metadata of uploaded resources, such that the teacher does not need to manually enter any tags.

Yactul: Continuous active learning & formative assessment

Yactul is a gamified student response platform for formative assessment both in the classroom and at home (Grévisse et al. 2017d). In class, activities can be used either after a concept has been explained to check the students’ understanding, or before a concept will be introduced, as an incentive. As opposed to similar systems such as Kahoot!, Yactul is based on an extensible architecture enabling the dynamic integration of further activity types. An initial set of activity types was presented in Grévisse et al. (2017d). Gamified experiences could potentially be designed for any domain, such as shown in Fig. 5.

Fig. 5

Blood Count Analysis. A Yactul activity type in the domain of life sciences. Source: (Gashi 2018)

Yactul extends the learning experience from the classroom to the private study environment of the individual student. The Yactul app lets users replay all activities played in class, at any time. As activities are tagged with concepts from, e.g., the ALMA Ontology, the app provides the user with formative assessment which keeps track of his performance on a per-concept basis. Schimberg (2017) developed a spaced repetition algorithm based on the SM-2 algorithm from the SuperMemo family, tailored to concept-based recommendation of activities. If activities on a given concept have been answered wrong in the past, they are more likely to be asked again, and the concept will be marked as less well understood.

Apart from the concept-based recommendation of activities, semantic tagging of Yactul activities also enables the retrieval of learning material on the concept of an activity from within the Yactul app. If a user experiences issues during an activity, he can indicate that he does not know the answer, which will be recorded for the spaced repetition algorithm. Subsequently, he can inspect a list of related resources indexed on the ALMA repository, as shown in Fig. 6.

Fig. 6

Learning Material Retrieval in the Yactul App. A list of resources concerning the concept For Loop is shown

As will be explained in the subsequent section, it is not only possible for Yactul to integrate related learning material, but Yactul activities themselves can be considered as a type of (active) learning material and be integrated in another, potentially non-gamified context, as showcased in Fig. 7.

Fig. 7

Example use case of SoLeMiO. Here, a Yactul activity related to the concept of loops can be played from within PowerPoint. Source: (Grévisse et al. 2018b)

Authoring support

As the general assumption of the ALMA-Yactul ecosystem is that learning material is semantically annotated at the level of its metadata, proper annotation support for authors should be provided. This support should neither require substantial technical skills, nor force authors of learning resources to use tools different from those they are used to. The ecosystem comprises annotation support for different tools, such as a plugin for Adobe Acrobat or macros for LATE X-based documents. When creating Yactul activities, authors also need to append concepts in order to relate learning material to them as well as make them relatable as an active learning resource.

To avoid a significant increase in the time necessary to author a resource caused by the annotation process, the teacher should be provided a semi-automatic annotation support. While fully automatic semantic annotation can be incomplete or incorrect, semi-automatic support suggests a set of initial concepts that the teacher can choose to add, while still giving him the possibility to add any missing one. Therefore, we developed SoLeMiO, an Add-in for Office 365 (Grévisse et al. 2018a). This popular suite is available at many educational institutions free of charge, and its modern add-in architecture enables their use on any platform the suite is available for. SoLeMiO can be used in Word and PowerPoint. Similar to the sketchnoting app, it uses concept recognition and entity linking tools to find and suggest concepts to annotate the resource with. From the recognized concepts, a semantic representation is built using additional information from Open Knowledge Graphs through expansion and filtering strategies. A detailed description of this process was presented in Grévisse et al. (2018a). Annotation can happen either at document level, or in a fine-grained way, assigned to a text selection in a Word document or a slide in PowerPoint. An initial performance evaluation of tag and resource recommendations yielded promising results. On average, relevant concepts as well as resources could be found within the first 3 suggestions.

Apart from the annotation functionality, SoLeMiO also enables the integration of related learning material within an Office product. By querying the ALMA repository with the concepts annotated in the document, a related knowledge base article, a map, a video, or even a Yactul activity (Fig. 7) can be shown in the task pane next to the document. Similar to the sketchnoting app, this allows students to get more information or a different explanation on the content of the main resource. An extensive discussion on the possible use cases for SoLeMiO in different domains has been described in Grévisse et al. (2018b).

Evaluation

Usage data

During the academic year 2018/2019, data on the usage of the ALMA-Yactul ecosystem was collected. University students were using different tools to retrieve learning material in their current study context. For instance, when trying to solve programming exercises, they could rely on the Eclipse plugin or the hashtags provided on lab sheets. Furthermore, they could replay the quizzes played in class in the Yactul app, which also enables users to retrieve resources corresponding to the topic(s) covered in a quiz question. The most popular resources were PDF documents on basic Java programming, concerning console output, conditions and loops. In total, 1460 resource visits have been performed, including the Yactul app. On average, a resource has been visited 7 times. As can be seen in Fig. 8, the Eclipse plugin was used significantly less than the hashtags on lab sheets or the Yactul app. This can be explained by the fact that the Eclipse plugin provides scaffolding support for already written code, whereas the need for learning material might actually be much higher when being confronted with an exercise that needs to be written from scratch. It could be determined that the exercise causing most retrievals (152) of learning material was performed on the first exercise introducing basic OOP concepts, such as classes, methods and attributes, in Java. The collected data can give teachers an intuition into which concepts required more attention from students or which exercises were particularly challenging to them. The Yactul app has been installed by around 150 students. 3 out of 5 users have retrieved learning material from within this app, with an average of 7 resources per user. As a timestamp is collected on each resource visit, we could see local peaks in activity before each exam period.

Fig. 8

Learning Material Retrieval by Tool

Survey among university students

At the end of summer semester 2019, a survey (Fig. 9) was conducted among N=11 students from a continuing education bachelor in computer science taught to professionally active adults, 1 woman and 10 men, from 28 to 48 years (average: 36). 8 out of the 11 subjects indicated to have had previous experience in computer programming before beginning their university studies. Subjects agreed that the search for learning material causes an additional mental effort and were in favor of reducing this effort by using tools that helped them pinpoint relevant learning material. Subjects highly appreciate playing quiz activities during the learning process. They indicated they have often played quizzes in the Yactul app and sometimes retrieved learning material from it. The app was considered very easy and useful to play quizzes and to retrieve learning material. Regarding what participants have most liked in the ALMA-Yactul ecosystem, “quiz” or “Yactul” was mentioned several times. One participant stated that "it helps you get a feeling if you know a subject or if you have to invest some more time". Subjects have indicated to rarely retrieve learning material using the Eclipse plugin, but considered the hashtags on lab sheets very easy to use and also useful. When asked which of the three tools helped students most in the process of retrieving learning material, 73% indicated the Yactul app, the rest opted for hashtags on lab sheets. Subjects agreed that tools from the ALMA-Yactul ecosystem helped them understand concepts during the learning process and solve programming tasks. 73% stated to have used them at home.

Fig. 9

Survey among university students

Experiment with high school students

While the ALMA-Yactul ecosystem was used in university courses where students were introduced to concepts in class and thereby gained knowledge along their use of its tools, it would also be interesting to see whether its scaffolding support was also beneficial to subjects without prior knowledge and without accompanying course. For this purpose, an experiment was carried out with high school students with little to no prior knowledge in computer programming.

Population

The experiment was conducted in two Luxembourgish high schools. In the Lycée Hubert Clément Esch-sur-Alzette (LHCE), n_LHCE=12 pupils from the mathematics track participated, 4 girls and 8 boys, between 16 and 18 years old (M=16.9, SD=0.7). Apart from one boy who indicated to have a little experience in computer programming, all other 11 pupils did not have any prior knowledge in this domain. Furthermore, pupils from the Lycée Bel-Val (LBV) from a social sciences track participated. Among the n_LBV=19 pupils, there were 10 girls and 9 boys, between 17 and 23 years old (M=19.8, SD=1.3). There were 6 pupils (32%) indicating to have had previous experience in programming, as they were previously on a different track comprising programming classes.

Experimental setup

The experiment, called Homer’s WhatsApp, focussed on the approach of using hashtags to integrate learning material in tasks. The frame story was to help Homer Simpson recover the encryption key of his WhatsApp chat. Using a website that emulated WhatsApp, shown in Fig. 10, subjects were to enter the encryption key, which was the concatenation of the answer keys of 5 Java programming-related tasks. Without the correct key, responses by Homer were unclear, resulting in text being replaced with Japanese symbols and images being noisy. The more correct the entered key became, the clearer became his answers. Subjects could intent as many times as they wished. To limit the effects of subjects trying to game the system, the algorithm determining the quality of communication only slowly yielded better communication. In addition, the correct answer codes were relatively long, making it difficult to try different permutations.

Fig. 10

Interface of Homer’s WhatsApp

The population was split into a test group (50%) and a control group (50%). While members of the test group were given questionnaires with hashtags next to the tasks, similar to lab sheets at the university, and could use the ALMA repository to find related learning material, members of the control group were neither provided with hashtags, nor knew of the existence of the ALMA repository. Instead, the latter had to use a traditional search engine. The scaffolding approach in the test group could be considered as a guided form of discovery learning.

The problem-solving tasks were of low to intermediate difficulty for novice programming students. As shown in Fig. 11, tasks proposed some already written code. The first task consisted in assigning the correct data type to variables. The second one required students to sort code snippets such that conditional statements would reflect exemplary program executions. In task 3, the correct operator among 4 possibilities had to be chosen, two of which (= and ==) being very similar but have substantially different outcomes. Task 4 required students to understand a piece of code, of nested loops, in order to choose the correct outcome among 4 alternatives. A non-obvious detail - one of the loops saw his counter incremented by 2 at each iteration - with particular importance to the output increased the difficulty level of this task. Similar to task 2, task 5 was again about sorting code snippets in order to reach a certain outcome. While the tasks did not expect subjects to write syntactically correct code, their problem-solving nature could still verify whether they understood, to a certain extent, the concepts of conditional statements or loops, by letting them sort the code snippets to match the proposed output or select the correct output from the given code.

Fig. 11

Example of a problem-solving task in the high school experiment

Research questions

The main research questions to be studied in this experiment were the following:

• Is there a significant performance difference in correctly resolving the problem-solving tasks between the test and control group?

• Are there significant differences between genders or classes?

• What is the perceived usefulness of the ALMA approach (for members of the test group)?

• What is the perceived mental effort while solving the tasks?

Results

The number of correct answers by task for both test and control group is shown in Fig. 12. As a statistical test for determining any significant difference between gender or group, Fisher’s exact test was used, which is considered to perform well on small contingency tables. No significant difference between classes could be determined. The best-performing tasks were 1 and 3 at LHCE and LBV, respectively. Task 2 was the least well-performing task in both schools, yet subjects in the test group overall performed significantly better than the control group (p<0.01). In fact, no member of the control group in any of the two classes was able to correctly answer this question. Task 4 was the second least well-performing task at both schools. Interestingly, this task presented a significant difference with respect to gender (p<0.05). In fact, only one boy at LBV answered it correctly. Most of the time, the previously mentioned subtlety, i.e., updating the outer loop by 2 instead of 1 after each iteration, was overseen. Task 5 was of neutral success.

Fig. 12

Number of correct answers by task. Apart from task 4, the test group was outperforming the control group. In case of task 2, it was significantly doing so (p<0.01)

Figure 13 shows the total mark distribution for both test and control groups. Here, Student’s t-test is used for determining any significant differences. As an effect size measure, Cohen’s d (Cohen 1988) is used. There was no significant difference between genders or between classes. The average total mark was M=2.42(SD=1.41). The p-value of 0.056 for the t-test between groups is close to the usual threshold of α=0.05, yet not significant. However, as previously mentioned, 6 students at LBV already had previous experience in programming, coming from a different branch. While their performance was rather neutral (M=3.17, SD=0.98), they were significantly better performing than subjects without prior knowledge (p=0.015, d=1.31). By excluding subjects with previous experience, a significant difference of mean total marks could indeed be noted (p=0.042, d=0.898). Cohen’s d indicates a large effect size. In fact, the test group excluding students with previous knowledge achieved a mean total mark of 2.83 out of 5 (SD=1.75), whereas the corresponding control group only achieved a mean total mark of 1.58 out of 5 (SD=0.90).

Fig. 13

Total marks for test and control group. By excluding individuals with previous experience, the outperformance of the test group becomes even more evident

Discussion

There was a significant difference between test and control group in the overall performance, and, in particular, for task 2. The use of hashtags and the integration of learning material using the ALMA repository among subjects without prior knowledge was thus beneficial, compared to their colleagues using a traditional search engine. This finding is similar to (García-González et al. 2017), in which the test group could benefit from a similar scaffolding support provided in the Sakai LMS. The choice of excluding subjects with prior knowledge is valid, as the experiment was explicitly aimed at subjects without prior knowledge. Apart from task 4, there was no significant difference between genders. There was none between the two school classes, either. However, while at LHCE, there were 3 girls from the test group who solved all 5 tasks, no participant at LBV was able to reach more than 4 correct tasks. The first girl to solve the problem was done with the tasks within 48 min, according to the data collected by the WhatsApp-like website. This is an impressing result, considering the lack of previous knowledge in the domain of computer programming and the complexity of (some of) the tasks. At LBV, it was noted that the trial-and-error period set in after roughly half an hour, as the provided key length was continuously the one of the solution, yet nobody was able to determine the correct key. However, after 50 minutes into the experiment, one participant was one character off the correct solution.

After the problem-solving tasks, subjects were asked to fill out a survey on their opinion about the general search process for learning material and the tasks of the experiment. In addition, subjects in the test group were asked about their opinion regarding the ALMA approach. Answers were given on a 5-point Likert scale. Results are similar to the survey conducted at the university with adult learners.

Pupils mostly agreed that the search for learning material is an additional mental effort and would like to use tools to help them quickly find relevant learning material and thereby reduce the time and effort spent on the search process (Fig. 14). While they want to find a quick answer, to the majority it is important to understand in detail what a learning resource explains. The time less spent on the search could thus be invested in better understanding a suggested resource.

Fig. 14

Survey among high school students

While there could not be determined any significant differences between the two schools, members of the respective test groups were more or less enthusiastic about the scaffolding support provided by the ALMA approach (Fig. 15). Pupils at LHCE found that the linking between a task and the ALMA repository through hashtags is highly useful and rated their overall experience very positive. Also, they would rather recommend its use in their school. In general, subjects were slightly positive about the ALMA repository reducing the mental effort compared to a manual search. While tasks were considered rather difficult, students at LBV considered them significantly more difficult than their colleagues at LHCE (p<0.01, d=0.58, Fig. 16), and were also more stressed.

Fig. 15

Differences in the opinion about the ALMA approach between the two test groups

Fig. 16

Level of difficulty and stress perceived during the experiment

Asked about what features or aspects of the ALMA repository they especially liked, most answers were related to the benefits of using hashtags to quickly find related learning material, how easy it is to pinpoint relevant resources of the preferred type, and not being overwhelmed by the number of results. "I liked the fact that we could use hashtags. It made everything easier to find." stated one girl at LBV.

In semi-structured interviews with two pupils from the control groups, one girl mentioned that she considered the experiment as a fun challenge, a game she tried to succeed in as good as possible. She especially liked the gamification character of the frame story (Homer’s WhatsApp). The other pupil stated that she did not know what to search for, which is basically the definition of Meno’s paradox of inquiry. It helped her that the image returned by Homer changed favorably when giving correct answers. She consulted random pages returned by the search on Google, without knowing which page could be a reference. They were then introduced to the scaffolding support their fellow pupils from the test group could benefit from during the experiment. The interviewee from LBV redid the experiment, this time with the hashtags on the questionnaire leading to the ALMA repository. Her general impression of the ALMA approach was that it is very easy to use, very useful, that much of the mental effort of finding resources was reduced, that her overall experience with the ALMA repository was good and that she would definitely recommend it to be used in her school, including in human sciences subjects, such as pedagogy, psychology or philosophy, where text analysis often overwhelms learners with a myriad of information. She especially liked the use of hashtags, as "we are currently living in a hashtag world". When asked whether the ALMA approach would have helped her if she had not had any prior knowledge, she strongly affirmed, stating that "it is practically a course".

Triple E framework

There exist several technology integration models, such as TPACK or SAMR. However, these conceptual frameworks often lack a practical tool to evaluate the connection between integrated technology and learning goals. The Triple E Framework recently proposed by Liz Kolb was developed to evaluate the use of technology in educational settings by measuring along the axes engagement, enhancement and extension (Kolb 2017). To determine whether the technology could have a positive impact on student learning goals, the framework provides the measurement tool shown in Table 1. According to Kolb, the resulting 12 points can be interpreted as a "strong connection between technology tools and student’s take up of the learning goals".

Table 1 Evaluation of the ALMA-Yactul Ecosystem according to the Triple E Framework Measurement Tool

Conclusion & future work

This paper presented the ALMA-Yactul ecosystem, an approach for providing scaffolding support to learners by integrating related learning material in their current study context. Based on semantic annotations, this integration has been showcased for different applications and domains, such as a plugin for an IDE or an enhanced sketchnoting app. Authors of learning material are also provided with a semi-automatic annotation support in the popular Office 365 suite. Apart from formative assessment through spaced repetition, the Yactul app also benefits from integrating resources related to quiz activities. Our evaluation has shown that the ecosystem is actively used and appreciated among university students. An experiment with high school students lacking prior knowledge in the domain of computer programming showed a significant difference in performance compared to a traditional learning material search process. Finally, according to the Triple E Framework, our ecosystem performs well in the categories of engagement and enhancement.

The approach taken by our ecosystem can be used in direct instruction, enhancing traditional lectures, or in a self-directed, inquiry-based setting, where students can discover a domain through the scaffolding support. Still, even in a partially self-directed environment, it is appropriate to check the acquisition of fact-based knowledge through (self-)assessment, such as provided by Yactul.

There are several directions for future work. To further enhance the interoperability with respect to learning analytics across different systems, our ecosystem could benefit from the recent Experience API (xAPI) in order to track the progress of a learner in so-called Learning Record Stores (LRS). The enhanced sketchnoting app could be gamified by introducing Augmented Reality (AR) elements as learning resources. For instance, a history class could be enhanced by showing the process of a battle, mentioned in the notes, with AR-animated characters. Smart classroom environments with ambient intelligence, such as presented in Ntagianta et al. (2018), can understand the progress, feelings and attention level of students. In such a setting, the ALMA-Yactul framework could benefit from the gathered sensor data to suggest learners aid when struggling. For instance, if a pupil is gazing for a certain time at a task without being able to solve it, related resources could be provided. Finally, while the current student evaluation focussed on the use of hashtags to provide the scaffolding support, a user study on the sketchnoting app as well as a larger evaluation of SoLeMiO could provide further valuable insights.