Competency-based personalization for massive online learning

Scenario models for learning contexts

Figure 1 shows a simple scenario model. There are four activities or tasks (ovals), two actors (a professor and a student) and some resources (I/P linked) that are input to the activities or produced by the actor responsible for the activity (identified by a R link). Each activity is decomposed into sub-models (not shown on the figure), which describe it more precisely on one or more levels. This scenario will serve to illustrate the concepts presented in this paper.

In the first activity (Start), the student reads the general assignment for the scenario and the list of target competencies he/she is supposed to acquire. In the second one, using the information in a document called “Planet Properties”, the student builds a table of planet properties that is validated by the professor (in a MOOC, this should be automatically done by a software agent). In the third activity, using a version of this “Validated table”, he compares properties of planets to find out relations between them, and writes a text, “Validated relations”, to present his findings. In the last activity he is asked to order planets according to their distance to the Sun and to write his ideas on planets that can sustain life.

On the right side of the figure, three agents (i.e. recommenders or advisor agents) have been inserted to assist each student in executing the corresponding activity, by providing personalized advice. These agents, further explained in section “Recommendations based on competency comparison”, are also used to update each learner’s competency annotations with newly acquired competencies.

Semantic referencing of scenario components

As we have pointed out in (Paquette and Marino 2011), educational modeling languages and standards such as IMS-LD (2003) need to be improved with a structured knowledge and competency representation, in order to add useful semantic annotations to scenario components.

For semantic annotation, two main methods are generally used: annotation with concepts belonging to a taxonomy or annotation with natural language statements defining prerequisites and learning objectives.

We proposed to use a third approach based on competency referencing (Paquette 2007), where a competency can be understood as a skill applied to a knowledge, with a certain level of performance; the knowledge element in this definition being defined in a domain ontology.

A domain ontology needed for semantic annotations can be downloaded from ontology repositories available on the Web for various subjects. Only if the teacher/designer cannot find a suited one must he build it. Ontologies are “shared visualizations” of a knowledge domain, thus largely independent from a particular teacher’s view. What are specific are the skill and performance levels added to ontology components to define target competencies.

Unlike other approaches based on ontologies, such as OWL-OLM (Denaux et al. 2005) or Personal Reader (Dolog et al. 2004a), our model generalizes taxonomy-based annotations with OWL-DL ontologies annotations and adds mastery levels (generic skills and performance levels) to ontology references.

Furthermore, to state only that a person has to “know” a concept is an ambiguous statement. It does not say what exactly the person is able to do with the concept. For example, stating that someone “knows a certain device” may mean competencies ranging from “being able to describe its structure” to “being able to recognize its malfunction” to “being able to repair it”. Also, it is very different if a diagnosis is to be made in a familiar or novel situation, or with or without help; these are examples of performance indicators or criteria adding precision to the statement of a generic skill.

We thus define each competency as a triple (K, S, P) where K is a knowledge element, a class, a property or an individual from a domain ontology, S is a generic skill (a verb) from a taxonomy of skills, and P is the result of combining performance criteria values. Domain ontologies follow the W3C OWL-DL standard. The taxonomy of skills is simplified to a 10-level scale (0-PayAttention, 1-Memorize, 2-Explicitate, 3-Transpose, 4-Apply, 5-Analyze, 6-Repair, 7-Synthetize, 8-Evaluate, 9-Self-Control). The performance part is a combination of performance criteria values providing four performance levels (0.2-aware, 0.4-familiar, 0.6-productive, 0.8-expert), added to the skill level.

We have developed a Competency editor inside the TELOS tool set, to create and manipulate this kind of competency models. To connect competencies from a domain specific competency model with a learning scenario relating to this competency model, we developed another TELOS tool, called the Competency referencer. Using this tool, one can specify learners’ actual competencies, activities’ and resources’ entry and target competencies.

For example, using a domain ontology for solar system planets such as the one on Figure 2 and a competency model based on this ontology, competencies can be associated to resources from the scenario (as shown on Figure 3). The entry and target competencies describing such a resource, (“Planet Properties”), could be compared with the actual competencies of a user to verify if he has all of them, or some, or none, in his competency model, and offer a recommendation, such as to skip the resource (too easy for this user) or to study a preliminary document (this user doesn’t have the competences needed to study it, that is this resource resource entry competencies).

Activities, resources and user competency models

Selected components of a scenario are thus referenced using comparable competencies, based on the same domain ontologies and competency model. Resources and activities in a scenario are referenced by two sets of competencies, one for prerequisite competencies, and the other, for target competencies (i.e. learning objectives).

Figure 4 shows an example of a user portfolio that contains a list of competencies from two domains, instructional design and solar system planets. On the right side of the figure, evidences for the selected competency are shown as well as possible annotation or comments from an evaluator.

Knowledge and competency comparison relations

Consider two competencies C₁ = (K₁, S₁, P₁) and C₂ = (K₂, S₂, P₂). It will be rarely the case that the three parts will coincide, but we can evaluate the semantic proximity or nearness between C₁ and C₂, based on the respective positions of their knowledge parts in the ontology and the levels associated with the skills and the performance parts.

Our recommendation/adaptation agents will evaluate if a user’s actual competency coincides, is very near, near or far from the prerequisite or target competencies of a resource or an activity, as well as compared to the actual competencies of another user. These agents can also evaluate if a competency is stronger or weaker than another one according to the levels of its skill and performance parts. Or it can determine if the competency is more specific or more general according to the positions in the ontology of the corresponding knowledge components.

Thus, to take advantage of the competency model, we need to establish a formal framework for the evaluation of the proximity, strength or generality of competencies. In the next section we define the semantic proximity between knowledge parts of a competency. In section “Semantic relationships between Competencies” we extend the comparison method to competencies by considering skills and performance.

Semantic proximity of the knowledge components

In this section, we focus only on the knowledge part of the competencies to be compared. Maidel et al. (2008), propose an approach in which taxonomies are exploited. Five different cases of matches between a concept A in the resource (or item) profile and a concept B in the user profile are considered. Various matching scores are given when a concept A in the item profile, a) is the same, b) is a parent, c) is a child, d) is a grandparent or e) is a grandchild of a concept in the user profile. Then, a similarity function is used to combine these scores to recommend resources to a user according to his acquired concepts. Maidel et al. (2008) state that if the use of a taxonomy is not considered, the recommendation quality significantly drops.

We agree with this statement but we believe that extending the comparison to ontology components instead of only specialization relations between classes would provide a stronger basis for competency comparison. Note for example that in the solar system domain, property comparisons are more important than class relationships.

We thus propose to define the semantic proximity between knowledge elements, based on their situation in the domain ontology. Knowledge references are components from OWL-DL ontologies that describe the knowledge in a resource. A few examples of these knowledge references are shown on Figure 2 that presents part of an ontology for solar system planets^a. Knowledge references can take six different forms: SolarSystemPlanet is a class reference (C); Neptune is an instance reference (I); SolarSystemPlanet/hasAtmosphere/Atmosphere is an object property reference with its domain and range classes (D-oP-R); Earth/hasSatellite/Moon is an object property instance reference (I-oP-I’); SolarSystemPlanet/hasOrbitalPeriod is a data property reference with its domain class (D-dP); Earth/hasNumberOfSatellites is a data property instance reference (I-dP).

We have investigated systematically these six forms of OWL-DL references to decide on the nearness of two references K1 and K2. For example, a concept (form C) is near its sub classes, super classes, and instances. It is also near an object or data property (forms D-oP-R and D-oP) that has a domain or range identical or equivalent to this concept. A property reference, with its domain and range (form D-oP-R) is near a sub-property or super-property with the same domain and range. It is also near to a sub-property or a super-property with a subclass or superclass of its domain and range.

Another comparison criteria is concerned with a reference K₁ being more general or more specific than another one K₂. For example, K₁ is more general than K₂ if K₁ is a superclass of K₂, or has K₂ as an instance, or appears as domain or range of a data or object property reference K₂, or contains an instance in the domain or range of a data or object property instance reference K₂.

Semantic relationships between competencies

We now extend the comparison to competencies, adding the skill (S) and performance (P) components of the competency model. Figure 5 presents a few comparison cases between two competencies C₁ = (K₁, S₁, P₁) and C₂ = (K₂, S₂, P₂) in the case where K₁ is near K_2. Other cases are not considered, i.e. comparison fails.

To illustrate the heuristics, the (S, P) couples are represented on a 2-dimensional scale in Figure 5. Skills are ordered from 0 to 9 and grouped into four classes as follows: . Performance indicators are grouped into four decimal levels.

For example, a competency C1 with an analyze skill at an expert level is labeled 5.8 (S₁ + P₁). A competency C2 at a level 7.2 or 6.4 will be considered near and stronger than C1 because the synthesize skill or the repair skill are in the same class than the analyze skill, but one or two levels higher in the skill’s hierarchy. On the other hand, a competency C2 at a level 5.2 will be considered very near and weaker than C1 because it has the same skill’s level but with a lower performance level. Other possible competencies in the “far zone” will be considered too far to be comparable.

Also, depending on the relationship between K1 and K2, C2 will be defined as equivalent, more general or more specific than C1. These relations between competencies can also be combined to define more complex relationships. For instance, it is possible for a competency reference to be at the same time identical or very near/near, stronger/weaker and more general/more specific than another one.

We have developed an inference system that deduces these relations from a competency referential edited by a competency editor as presented above.

Competency-based conditions and rules

Recommendation agents are added to a scenario, linked to some of the activities or functions (grouping of activities) called insertion points, as shown in Figure 1. The designer defines these agents by creating a set of rules. In each rule, one and only one of the actors linked (using R links) to the activity at the insertion point is chosen as the receiver of the recommendation. If a triggering event occurs at run time such as “activity completed”, “resource opened”, etc., each applicable rule condition is evaluated and its actions are triggered or not, depending on the evaluation of the condition.

When this condition is evaluated, competencies in the user’s model, for the specified user at the insertion point, are retrieved, together with the list of target competencies for the resource “Essay”. The evaluation of the relation “Near and MoreSpecific” provides a true or false value according to the method exposed in section “Semantic relationships between Competencies”.

Recommendation actions

The action part of an agent’s rule can perform a number of recommendations or adaptations in a scenario: give advice to an actor, notify another actor than the one targeted by the rule, recommend various learning resources, update the user’s model, and propose to jump to another activity or to another learning scenario.

All these possibilities have been implemented in TELOS. On Figure 1, we have presented a simple scenario with three recommendation agents. For example, Recommender agent #1 on Figure 1 will verify if the learner has succeeded the second activity in the scenario (“Build a table…”). It has 3 rules, shown on the screen-shot of Figure 6: update model, notify instructor or advice to learner.

The rule “Update User Model” transfers the list of target competencies associated to the activity to the user’s model, if he has succeeded the activity, and built a validated table of planet properties. If he has failed, the second rule will send a notification to the professor to interact with the user. Finally, a third rule (selected on Figure 6) provides an advice to a user that has partly succeeded the activity, recommending the resource shown on the figure.

Multi-agent recommendation system

The recommendation system is epiphyte to the learning system, implying that it can evolve and change without modifying the learning scenario. One can even have various different recommendation systems for a same learning scenario, possibly for teachers of the same course with different pedagogies. The system is based on production rules of the form < actor, event, condition, action>. A particular action “delegate” allows the propagation of an local event and its activation context to other scenario parts, for instance to adapt the final exam content, based on a particular activity performance.

The MOOC global scenario

Figure 7 present a MOOC scenario for a course on interviewing techniques to which a preliminary module has been added for personnalization. Each of the five modules (represented by single white ovals) spans on 1–2 weeks.

• Module 0 proposes preliminary activities evaluated using various methods to subdivide the group of learners into subgroups having similar competencies; e.g. three subgroups termed “novice”, “intermediate” or “advanced”. There could be more subgroups but three seems sufficient for our illustration purpose.

• When the subgroups are created, each learner will afterwards (P link) do an activity (represented by multiple ovals) where he/she will choose a plan for the rest of the course adapted to his/her subgroups’ competency profile. This plan is produced by an advisory agent that will propose an ordering of modules A, B and C, according to the results of module 0. For the advanced subgroup for example, module B can be skipped; for the intermediate subgroup, module B is proposed after module C as enrichment; for the novice group, module A, B and C are done in that sequential order.

• For all three subgroups, module D follows the other modules completion.

• For each of module A, B, C and D, the activities and the resources proposed to learners will differ from one subgroup to the other, according to the recommendations of the corresponding advisory agents.

• Another actor, the “Community Animator”, participates in module 0 and also triggers the operation that sends automatic attestations to learners, after module D is completed.

Creating the subgroups

We will now expand on Figure 8 the sub-model for module 0 where the subgroups are created to help personalize the learning scenario. This module starts when the community animator sends a notification to the learners to start the module’s first activity. Before that, the community animator will have produced a self-evaluation questionnaire to be used by learners to assess their own competencies. For this, he triggers the Q&A automatic production operation of Figure 8. This operation takes as input a Course competency profile such as the one shown on Figure 9 and produces a Competency self-evaluation questionnaire. This questionnaire presents each competency in the profile to each learner and asks if he/she considers him/herself at one of these four levels: aware, familiar, productive, or advanced. These levels are defined according to the same competency performance criteria shown on Figure 9 and the definition of the four levels are provided to the learners to help them select the right level that corresponds to their situation.

This questionnaire will be used in the third activity of module 0. In the first activity, each learner consults video and/or texts on interviewing techniques. Afterwards, they pass a Q&A test on interviewing techniques, and the results are compiled automatically for each learner. MOOC platforms usually provide such operations to assess learners.

The second activity engages learners to write an interview plan. The input is a Problem statement where an organization aims to recruit new professionals to achieve strategic functions within the organization. The Produced document will be automatically and randomly distributed to a number of peer learners who will rate the documents assigned to them using a questionnaire related to the competency profile for the course. The Plan evaluation results of the second activity will be compiled by the operation Distribute to peer learners and compile ratings shown of Figure 8.

In the third activity, each learner will assess his/her own competencies using the Competency self-evaluation questionnaire prepared by the course designer as mentioned before, therefore producing the Competencies evaluation results shown on Figure 8.

Finally, an operation for “Automatic classification in subgroups” will combine the results data of the three activities, according to some weighting policy established by the course designer, producing the list of learners for each subgroup and sending a notification to each learner to inform them on their assigned subgroup.

Then the execution of module 0 stops and the flow of control in the scenario moves back to the global scenario and to the other activities and modules of the MOOC.

Competency referencing of actors, modules and activities, and resources

Competency referencing plays a major role in the proposed personalization process. To start the process, a specific competency profile or model must be built according to the course content.

Such a profile, named here Interviewing competencies is built using the TELOS competency editor shown on Figure 9. Other competency profiles have been built for other course subject. Each profile follows the model presented in sections “Semantic referencing of scenario components” and “Activities, resources and user competency models”. First the knowledge part is declared, then a generic skill is selected with its type (or meta-domain) and finally the performance criteria are set, combined automatically to produce the performance level for each competency.

The Interviewing competencies’ profile is shown on the left part of Figure 9. One of the target competencies for the course labeled “1c-Select an adequate interview process” is selected for edition and displayed on the right side of the figure. Its knowledge part is a property of an interview process with its domain and co-domain classes; its generic skill part is level 2-Explicitate and is typed “cognitive”; its performance part is at the level 6-productive resulting from the combination of the five criteria below. These competency parts are combined in a precise natural language statement: Define the best type of interview process that allows an efficient evaluation of desired skills, without help, in any situation.

This competency profile has four sections corresponding to the various modules of the course. These modules are referenced by the competencies in corresponding sections of the profile; for example, module A is referenced by all the competencies in section “Preparing for the interview”. Inside module A, the activities and resources are referenced by only part of the competencies for the module. It is up to the designer to ensure that all the competencies of the module are well covered and well distributed amongst the activities and the resources that compose the module.

The learner’s competencies are set initially at the end of the preliminary module 0. At the end of this module, initial competencies have been attributed to the different learners during the classification operation that has produced the subgroups of learners. These initial competencies of individual learners will evolve throughout the course scenario.

Consider an activity in module A that consists in selecting an adequate interview process. After evaluation of this activity, depending on the quality of the outcome, a competency similar to 1c, possibly with a lower performance level will be added to the learner’s model in a similar way as explained in section “Multi-agent recommendation system”. Also, an improvement activity will be proposed by an advisory agent to this learner by providing additional resources to consult. Hopefully, at the end of the course, the set of actual competencies of this learner will reach the targeted levels in the competency profile for the course.

The adaptation/personalization process

We now focus on the work of the design team. Figure 10 presents an abstracted process from the example of the previous section. It starts with the “Original scenario” and leads to the production of an “Adapted scenario”.

In a first activity, the design team will prepare a competency profile using a competency editor. The TELOS competency editor presented on Figure 9 is one solution that provides a natural language statement of the competencies, but more important it provides their underlying structure (K, S, P). Simpler solutions exist such as an outline/tree editor that is available in any text editor, but it would have to be extended to provide the structural parts of a competency on which competency comparison relations can be computed. One could also use a simple taxonomy of concepts instead of a complete OWL ontology as we have done in the examples here.

In the second activity we need to reference the MOOC modules with target comptencies from the profile defined in the first activity. For this we need a semantic referencing tool to assign target competency at least to each module of the course. The same tool would also enable designers to assign entry competencies to modules in order to notify learners who would not have the capacity to start a module and to recommend preparatory activities in this case. Inside the module, the design team would also annotate with competencies critical activities and resources for which recommendations seem necessary.

The next two functions on Figure 10 represent the two methods of adaptation employed in the example of section “Personalization of massive open online courses”. They are detailed in sub-graphs that we will discuss in the following paragraphs.

Adding a preliminary module for creating subgroups

The first adaptation method is adding a preliminary module containing one or more ways to evaluate the competencies of each learner. The evaluation results serve to define subgroups of learner with similar, compatible or complementary competencies, as well as provide a basis for identifying the competencies of individual learners. Figure 11 presents this sub-process.

The first design activity is to add a preliminary module to the original scenario and reference the learning activities and resources within that module with competencies as was done previously for the rest of the course. The new module will be created using a scenario editor. It can be the same editor used to build the original scenario or another one. For example, the new module could be built with Open-edX studio if the original scenario had been built in this editor. The new module could also be integrated in Open-edX by an external call to another scenario editor obtained by specializing the TELOS editor used in section “Personalization of massive open online courses”.

There are many options for the learning activities that can be inserted in the preliminary module to evaluate learners’ competencies. In the example of section “Personalization of massive open online courses”, we presented three main options that we generalize in the design sub-process of Figure 11.

The first one is compulsory. It consists in automatically producing a questionnaire from the competency profile to enable each learner to rate his own competencies for the course. This competency questionnaire can be very simple, for example a set of multiple-choice question based on the competency statements with choices such as {unaware, familiar, productive, advanced} or any other set of value consistent with the way competencies are defined. In this way, a distance between the actual competencies of learners and the target competencies for a module, activity or resource can be automatically computed for each learner. The same kind of competency comparison can be done to check if a learner has the prerequisite competencies for a module, activity or resource.

One might think it is not sufficient to have learners evaluate their own competencies. Some learners might overestimate or underestimate themselves. Still, there is an incentive for a learner to be as precise as possible because providing the best information possible opens the possibility for better assistance to the learner from the system.

But additional ways to assess competencies are offered with the two other options of Figure 11. In the first case, if video or text consultation activities are integrated to the preliminary module scenario, a simple Q&A test can be added, producing evaluation results automatically. In the second case, if there are production or construction activities (essays, problem solutions, graphs,…), the resulting productions can be distributed to peer learners who will rate a limited number of their peers by filling a small assessment questionnaire. In both cases, the Q&A and the peer assessment questionnaires must be related to the competency statements for the course, so that the results can be combined with the self-assessment results.

If two or more evaluation activities are inserted in the MOOC preliminary module, we need to add some automatic operations in the system, one of the features in TELOS. One operation simply combines the evaluation results using a weighting policy before adding the result in the learner’s competency references. Another one is the integration of a learner clustering operation. This important operation combines the various competency evaluation results using a clustering policy specified by the designers. The policy will include at least the relative weights assigned to the various evaluation methods, and a specified number of subgroups to be created. The tool will create automatically the lists of subgroup members and notify the individuals on their membership and their peers’ average competencies.

More advanced parameters can be included in the clustering policy. One way would be to integrate in the same group learners with similar cognitive competencies, but with complementary technical or socio-affective competencies, so that collaboration inside a group can be more productive. Another extension is to use historic data from previous instances of the course, using learning analytics methods, to cluster learners according to a clustering policy.

Inserting recommenders/advisory agents in the scenario

Finally, to conclude the adaptation process, the design team will use a scenario editor to insert recommenders or advisory agents for critical modules, activities or resources in the course scenario in a way similar to the previous sections.

Right after the preliminary module, when the subgroups have been created, the design team can insert a plan recommender agent to propose to each subgroup an adapted path within the course. For this, the initial scenario must be designed in a modular way so that the modules can be followed in different orders, and also the activities and resources within a module. Also options can be provided between modules and/or activities and resources. This leads to a number of alternative plans that can be proposed to the various subgroups.

Next, the design team will identify critical modules and activities where resource recommender agents are inserted. At these corresponding insertion point, additional or alternative resources can be proposed. Also, notification recommender agents can be added to propose a change in the study plan. As long as a learner succeeds in some activities, his actual competencies will evolve and he must be notified of the change. Also, sometimes, the evolution of a learner’s competencies will create a situation where he must be notified to switch to another subgroup plan.