Augmented intelligence in educational data mining

Abstract

Educational data mining (EDM) processes have shifted towards open-ended processes with visualizations and parameter and predictive model adjusting. Data and models in hyperdimensions can be visualized for end-users with popular data mining platforms such as Weka and RapidMiner. Multiple studies have shown how the adjusting and even creating the decision tree classifiers help EDM end-users to better comprehend the dataset and the context where the data has been collected. To harness the power of such open-ended approach in EDM, we introduce a novel Augmented Intelligence method and a cluster analysis algorithm Neural N-Tree. These contributions allow EDM end-users to analyze educational data in an iterative process where the knowledge discovery and the accuracy of the predictive model generated by the algorithm increases over time through the interactions between the models and the end-users. In contrast to other similar approaches, the key in our method is in the model adjusting and not in parameter tuning. We report a study where the potential EDM end-users clustered data from an education setting and interacted with Neural N-Tree models by following Augmented Intelligence method. The findings of the study suggest that the accuracy of the models evolve over time and especially the end-users who have a adequate level of knowledge from data mining benefit from the method. Moreover, the study indicates that the knowledge discovery is possible through AUI.

Introduction

Educational data mining (EDM) is a collection of methods where raw data sets collected from educational settings are transformed into meaningful presentations of the educational setting to guide the stakeholders related to the context (Baker and Yacef 2009). Such stakeholders include both the students and the teachers. The typical data mining approaches applied in EDM are classification (Pandey and Pal 2011), cluster analysis (Merceron and Yacef 2005a), outlier detection (Goyal and Vohra 2012), and association rule mining (Romero et al. 2008). Data collected from the educational settings can vary in size and in dimensions. For instance, the data collected from a MOOC (Massive Open Online Course) may contain thousands of data items in comparison to a dataset collected from making activities of less than 20 students. For the larger datasets, many data mining algorithms perform well but for the smaller samples, there are few algorithms that actually are able to fit to the data. The algorithms that can cope with smaller datasets include for instance support vector machines (Moghaddam and Yang 2000) for classification and hierarchical clustering algorithms for cluster analysis (Abbas 2008).

Algorithms used in the traditional data mining are also widely used in the EDM context (Sachin and Vijay 2012). The most used algorithms are associated with classification, cluster analysis, and relationship mining (Baker and Yacef 2009). Examples of such algorithms are decision tree classifiers and neural networks (Ahmed and Elaraby 2014) (Sachin and Vijay 2012). Another widely used technique in the EDM domain is cluster analysis. The commonly used cluster analysis methods in the EDM are k-means (Merceron and Yacef 2005b) and hierarchical cluster analysis algorithms (Antonenko et al. 2012). For the visualizations of the dataset, common dimensionality reduction methods such as self-organising maps and principal component analysis are typically used in cases where the dimensionality of the data vectors is more than three.

There are specifically developed tools available for EDM tasks but also general data mining and machine learning tools are commonly used in educational contexts. Among the most used tools are Weka (Hall et al. 2009) and RapidMiner (Hofmann and Klinkenberg 2013). Both of the tools offer extensive set of algorithms to conduct data mining processes such as cluster analysis and classification. Also, the current tools used for EDM usually operate in a white box manner and the end-users are free to explore and adjust the algorithms and the produced predictive models through several parameters and visualizations. The algorithms and the models produced by the algorithms can be customized in various ways. For instance in RapidMiner, the end-users can adjust hundreds of parameters that have an influence in the analysis outcomes and the process.

Augmented intelligence (AUI) is a term that describes systems that merge human and the artificial intelligence (AI) (Rouse and Spohrer 2018). In contrast to AI, AUI aims to enhance the human’s perception and support humans in decision making, learning or planning with help of AI, instead of replacing human agents in processes automated with AI. In this paper, we demonstrate how AUI can be used in the EDM context to increase the accuracy of predictive models generated by AI algorithms. The principles of this method were presented earlier by Jormanainen and Sutinen (Jormanainen and Sutinen 2013) who demonstrated that teachers benefit from an open and adjustable data mining process, which uses white-box data mining algorithms (Delibasic et al. 2013), such as decision trees. The fundamental requirement for an open EDM process is that the output of the selected algorithm is easy to visualize (as decision trees are (Ankerst et al. 1999)). The end-users can access the data and gain understanding from the data or the EDM process through the visualizations. Furthermore, the teacher has to be able to adjust the predictive model produced by the selected data mining approach, and also iterate the classifier constructing process as necessary (Jormanainen and Sutinen 2013).

In our research, the concept of AUI refers to an iterative process, where in each cycle of the process both the EDM end-user and the computer collectively apply their learning outcomes to produce the output together. In the AUI method, the learning of the machine has an influence to the end-user’s learning and vice versa. The similar understandable and interpretable approaches for machine learning have been explored in the field of explainable artificial intelligence (Gunning 2017). Yet, our method combine the learning of both a human actor and a computer in an EDM context, instead of trying to explicitly explain the learning of a machine and a human. Furthermore, the AUI method does not solely depend on the changing parameters that has an influence to the built model of a predictive algorithm. Instead, in the AUI method, the actual predictive model generated by the algorithm is adjustable.

Jormanainen and Sutinen (Jormanainen and Sutinen 2013) successfully investigated the effect of an open data mining process, but their study was limited for using decision trees only. Multiple platforms currently allow end-users to adjust or even build a decision tree classifier in the EMD context and the studies related to these platforms show that the end-users benefit from such processes (Delibasic et al. 2013; Kovanović et al. 2017). For a more generalizable AUI method in EDM, there is a need to expand the scope to other commonly used algorithms. However, for example providing an easily interpretable visualisation for an output of a clustering algorithm in hyper-dimensional space is not trivial. For the purposes of AUI method, we have developed a novel clustering algorithm called Neural N-Tree (Toivonen et al. 2018), which tackles some of the challenges appearing typically in educational data mining settings.

To explore the feasibility of our concept of the AUI, we organised a set of studies to answer to the following research questions.

• 1 Does the accuracy of a predictive model increase over time through Augmented Intelligence method?

• 2 Are there any indicators that the Augmented Intelligence method leads to the knowledge discovery?

To answer these research questions, we conducted two studies. In the first study, two groups of users completed a simple clustering task to explore a set of educational data. The second study involved teachers who used the Neural N-Tree and AUI method to analyze data collected from a classroom setting where they were teaching. The experiment group in the first study used a data mining tool implemented by following the AUI principles and the control group of the study used traditionally used cluster analysis algorithm k-means and also the accuracy of the k-means was recorded. Furthermore, we ran tests with RapidMiner to cluster the same dataset to study how another state-of-art EDM tool performs. However, our intention was not to compare the different cluster analysis algorithms but instead to set a benchmark to model adjusting process in order to get a perspective how widely used and studied algorithms and platforms perform. Both the experiment group and the control group answered to questionnaires where the experiences from the methods were discussed. In the second study, the teachers using Neural N-Tree and the AUI method were interviewed and asked to fill a questionnaire that tested the user experience and the level of knowledge discovery. Based on the results, we argue that by adjusting not only the parameters of a given algorithm but also the models and interacting with the model visualizations, EDM end-users benefit from the AUI method and AUI method leads to the knowledge discovery.

This paper is organised as follows. First, we introduce the background and key concepts behind the AUI method, including technical details of the Neural N-Tree algorithm that was developed for the AUI tools. Second, we present the research setting, data collections, analysis, and the findings from the empirical study. Finally we discuss the implications of the results and present the directions for the future research on the topic.

Background

In this chapter, we present the conceptual backgrounds of the AUI method in the light of the previous research in the EDM field. Furthermore, we present the deeper technical details of the novel cluster analysis algorithm Neural N-Tree that was developed specifically for the AUI.

Open educational data mining

One of the examples of the open approaches for educational data mining is presented by Jormanainen and Sutinen (2012), who concluded that opening a decision tree classification algorithm and the EDM process benefits the teachers. While it is not uncommon to use a decision tree as a classifier in an EDM setting (Al-Radaideh et al. 2006), the novelty of the research in Jormanainen and Sutinen (2012) was that the teachers were engaged in data mining process while teaching, starting from labelling training data for the selected algorithm.

The Open Monitoring Environment (OME) presented in Jormanainen and Sutinen (2012) allowed teachers to label data collected from an educational robotics classroom setting real-time. The labels of the dataset items were expressing the performance of the students in the classroom, as perceived by the teacher. At any point during teaching, the teacher was able to automatically build a J48-decision tree (Bhargava et al. 2013) classifier based on the labelled training set. After the decision tree was built, it was visualised to the teacher, and the rules were presented as if-else conditions expressing the branches of the decision tree. The system had a 2D map view to the classroom, and students’ performance was reflected realtime in the map as predicted by the newly built classifier. If the teacher was not satisfied with the classifier, he or she was able to re-iterate the process by labelling more data and constructing the classifier again. Alternatively, the teacher was able to modify directly the rules of the classifier.

In the research presented in Jormanainen and Sutinen (2012), the authors noticed that the open EDM process was beneficial for the teachers. The OME system provided otherwise unseen views to students’ progress, and it helped the teachers to design better intervention strategies. Even more importantly, the open data mining process with easily interpretable decision trees produced useful information about the students’ progress with relatively small datasets. Many school environment settings, for example in K12 education, involve smaller groups of learners and thus produce smaller sets of data. This brings problems for mainstream classifiers. In general, most of the the mainstream classifiers, such as deep learning methods and neural networks, are difficult to use in the EDM settings because they require high volumes of data to work properly and they often fail to build generalisations. On the other hand, variety of tasks in the EDM domain require more versatile tools than just simple decision tree classifiers. The AUI method provides principles for implementing a wider range of data mining methods in open and interpretable way. However, to meet the goal, there is a need to design also novel algorithms, such as Neural N-Tree (see “Neural N-Tree” section for more details).

Two the most used platforms for EDM, Weka and RapidMiner also aim on open the data mining process through visualizations and parameter tuning. For instance, RapidMiner offers an extensive suite of visualizations for cluster analysis. The end-users can view the centroid values and graphs, where the centroids of the clusters are plotted to 2 dimensional line graph or even the cluster tree where a decision tree is built from the cluster labels and the dataset automatically. However, adjusting the centroids manually or the rules of the generated cluster decision tree is not trivial. Furthermore, RapidMiner’s analysis can be customized with a visual programming-like drag and drop interface. Figures 1 and 2 show how RapidMiner opens the data mining process by providing visual feedback from the clusters in hyper-dimensions. Figure 3 illustrates a visual presentation of hierarchical clustering for a sample dataset through a dendogram. Kovanovic et al (Delibasic et al. 2013) built a platform where the EDM end-users may build a decision trees from the scratch by using a simple user interface. The studies of these platforms suggest that the open-ended approach to EDM helps the end-user to gain a deeper understanding from the data mining process and the data itself, but the tools are not very accessible for the end-users, nor do they contain features that open the whole data mining process as our proposed AUI-driven toolset does.

Fig. 1

Centroid chart of k-means in RapidMiner

Fig. 2

Cluster tree of k-means in RapidMiner

Fig. 3

Dendogram of hierarchical clustering in RapidMiner

Augmented intelligence method

Augmented Intelligence method (AUI) in the educational data mining (EDM) context uses the concept of augmented intelligence where the traditional artificial intelligence (AI) supports the decision making, learning or planning of a human. Moreover, the AUI merges the human and the machine intelligence to generate knowledge in a way where both AI and human add unique value to the process (Rouse and Spohrer 2018). Instead of a group of two or more learners, there is a end-user and a machine who together build a solution for a context-specific problem by communicating through a machine learning environment. A role of the end-user is to provide a dataset for the machine learning algorithm, which builds a model from the dataset (computer’s interpretation over the phenomenon under investigation). The model is open and can be adjusted by the EDM end-users. The role of the end-user is active and the aim is not just the experiment with the parameters such as the number of clusters or a learning rate that has an influence in how the given algorithm builds a model. Instead the aim is to adjust the model itself.

The aim of the AUI method is that the EDM end-users generates new knowledge about the educational context based on the gathered dataset and the contextually developed model. Based on the generated knowledge and his or her pedagogical knowledge, the end-user is able to adjust the model built by the computer. Based on end-user’s informed adjustments, the computer re-builds a new model with new knowledge and data it has. The process is repeated, and during each step of the cycle, new knowledge about the educational context and the phenomenon under exploration is produced. The AUI method process is described in Fig. 4.

Fig. 4

Augmented intelligence in the educational data mining context

The AUI process leads to a cyclic knowledge discovery process of both end-user and computer outputs of the predictive models from the computer that are adjusted by the end-user (Fig. 4). Together, the outputs deepen the previous knowledge of the end-user about the context where the datasets were collected. The knowledge discovery is the union of the perception of the machine learning algorithm, the domain knowledge of the end-user, such as data scientist or a teacher, and a combination of these perceptions (from the end-user’s perspective). As the end-user possesses subjective knowledge from the learning context, the knowledge discovery process is also subjective to the end-user and not necessarily generalisable to the other end-users, e.g. the other teachers.

Neural N-Tree

To expand the scope of the AUI method from the easily interpretable decision trees, there is a need to develop also novel computational methods to serve the variety of different data mining tasks in contemporary smart learning environments. The usually applied data mining methods easily fail to build generalisable results in these highly contextual educational settings because of low volumes and sparsity of data. The Neural N-Tree (Toivonen et al. 2018) algorithm for educational cluster analysis is a balanced binary tree with each node containing a point vector. The length of the point vector equals the length of the input vectors. Neural N-Tree is constructed recursively by initialising each point vector with random numbers ranging from the minimum value to the maximum value occurring in the analysed data set. First, the algorithm takes the number of clusters n and creates a vector which length is n×2−1 with random point vectors. Second, from the created vector, a balanced binary tree is constructed as presented in Algorithms 1 and 2.

After constructing the Neural N-Tree with random point vectors, each terminal node is indexed by traversing Neural N-Tree with post-order traversal and assigning the terminal nodes with different index. Training of the algorithm is divided into two phases. In the first phase, the best matching unit, BMU is searched from the terminal nodes by comparing random sample vector with each terminal node and by choosing the terminal node as a BMU with the maximum distance with the sample vector. As a distance function d, several different metrics, such as Manhattan distance, euclidean distance and cosine distance can be considered depending on the input dataset attributes and dimensions. The maximum distance node or the least similar node is chosen as the BMU in order to spread the point adjusting to cover the whole tree. If the most similar terminal node would be chosen instead, then several branches of the created tree would not gain any adjusting. The measurement of the similarity is done by comparing the distance between the input vector and the terminal nodes’ points. When the BMU has been found, BMU is updated to be the parent of the BMU until one of the children of the root node has been reached. Before each update, the level order (Algorithm 6) starting from the BMU is calculated, and for each point of the node from the level order calculate:

$$P_{v}(s + 1) = P_{v}(s) + f(\omega)\times\alpha\times(D(t) - P_{v}(s))) $$

where f(ω) is the current level of the sub-tree +1 divided by the height of the sub-tree +1, α is the learning rate, D(t) is the current input vector and P_v is the point vector of the node. The purpose of the backward training is to initially adjust the points of the node to correspond the vectors in the dataset and to spread the adjusting process to cover the whole Neural N-Tree. The algorithm for backward training is shown in Algorithm 3.

In the second phase of training the Neural N-Tree, the training starts from the root node and the next node of the input vector is either the left or the right child of the current node depending on the distance of the input vector and the points of the right and the left child (the node with the smallest is chosen). The input vector is thus traversed through the Neural N-Tree from the root node to the one of the terminal nodes. After each step, the level order (Algorithm 6) starting from the current node (except the root node) is calculated and the points of the level order sub-tree starting from the current node are updated. The purpose of the forward training process is to adjust the paths for the input vectors for the cluster analysis process. The forward training algorithm can be seen in Algorithm 4.

The training of Neural N-Tree is based on updating point vectors of the nodes during the traversal of Neural N-Tree. Besides the point of the current node, also the points of the sub-trees starting from the current node are updated. The sub-tree update enables similar clusters to be located near each others. Yet only the left or the right sub-tree of the root node will be updated during the traversal. After both forward and backward training are complete, the third phase is the clustering itself. The method of clustering is similar to forward training but unlike during the forward training, no updates of the points take place. The clustering process can be seen in Algorithm 5.

Research about the feasibility of the AUI method

In this section, we introduce an experiment that we conducted to study whether the Augmented Intelligence (AUI) method with the Neural N-Tree algorithm in an EDM environment provide benefits for the end-users. Our research had two main objectives. First, we tested whether the AUI method in the EDM environment would generate objectively more accurate results after the end-users have interacted with the model. Second, we tested if the AUI method helps knowledge discovery in the dataset. The knowledge discovery aspect was studied with questionnaires and interviews with the teachers who used AUI method to analyze datasets collected from the context.

Research design

During the first phase of the experiment, a group of computer science major students (n=14) were devised with a task to cluster an educational dataset as well as possible. The group was divided in two subgroups. The experimental group (Group A) used an EDM system with an implementation of Neural N-Tree algorithm and possibility to iterate the data mining process by following the AUI method. The control group (Group B) subjects were given an EDM tool with the well-known k-means clustering algorithm, which was working in a black box. The students in the Group A were able to adjust the models generated by the Neural N-Tree algorithm. The model adjusting in this context does not only mean changing the parameters such as the learning rate or the number of the clusters. The model adjusting implies to process where the node values (the point vectors) are changed in the generated models. The purpose of the study was not to compare k-means’ and Neural N-Tree’s absolute accuracy in the given dataset. Instead, the widely used algorithm k-means offers a benchmark for accuracy level that should be quite equal to the accuracy level of Neural N-Tree in order for Neural N-Tree to be usable in the future studies.

Both groups were using the same dataset containing activity records of 62 students while they were learning with an educational simulator (Vahdat et al. 2015). The features within the dataset vectors presented the scores of the students from the assignments and the final feature was the total score of all assignments. In total, there were 17 features and the length of the dataset was 62, with each of the vector representing the scores of an individual student. We chose the dataset because it was collected from an educational setting, the features in the dataset vectors were easy to explain to the students participating in our research, and the clusters produced from the dataset were linear and easier to comprehend. Finally, our initial tests in the preparation phase indicated that both k-means and Neural N-Tree algorithms produce rather similar results with the dataset when measured with silhouette index (see more detailed explanation later in this article). This metrics was used in the study, and finding an equal starting point for both algorithms was essential.

During the second phase of the experiment two secondary school teachers from a local school used the same platform as the Group A in the first phase. The EDM environment implemented Neural N-Tree algorithm to analyze data collected from their teachers’ own class rooms. The teachers were interviewed. After using the tool, the teachers were interviewed with six pre-defined questions. The interviews were recorded and the answers of the teachers were compared to find any similarities between them. After the interviews, four teachers used the platform to analyze data collected from their teaching sessions in making activities. The teachers were given a questionnaire with 1 - 5 Likert scale. The aim of the questionnaires and the interviews was to measure the effect of AUI method in knowledge discovery and the understanding of the dataset and the educational context. Table 1 illustrates the pahses of the research conducted for this paper.

Table 1 Phases of the study

Research phase 1

Data collection

The participants for the first phase were selected to study on voluntary basis. The participants were students in an elective robotics course at the University of Eastern Finland, School of Computing. The participants were mainly 3rd or 4th year computer science major students. Some of them had gained previous experience on machine learning or data mining topics from their earlier studies. The students were assigned randomly to the experiment group (Group A, white-box, n=8), and to the control group (Group B, black-box, n=6, where 1 student did not answers to the questionnaire). Then a short introduction to the study was given, alongside with a description of the datasets used in the study. During the introduction, we described how the algorithms work and what the dataset was about. We described to the students that the dataset was collected from an educational setting and that the features within the dataset were individual task scores from a test. Also, principles and purpose of clustering algorithms in machine learning were explained to ensure that everyone understood what they were supposed to do.

For the study, we developed a web-based application for both groups. The experiment group (Group A) used an interface where the Neural N-Tree was visualised to the user in real-time and the nodes of Neural N-Tree were adjustable. If a student changed the point values, a new result produced by the clustering algorithm was rendered to the student. The students were able to change also the number of the terminal nodes in the Neural N-Tree, learning rate, and the number of iterations used to train the algorithm but the main focus was in the point adjusting: the researches stressed that the Group A should not use much time to select parameters that have an influence to the initial Neural N-Tree model. Furthermore, the students were able to choose whether the algorithm used backward training at all. The distance metrics used by all of the students was euclidean distance defined as

$$d(q, p) = \sqrt{\sum\limits_{i = 1}^{n}(q_{i} - p_{i})^{2}} $$

where q_i is the point of an input vector and p_i is the corresponding point of the node during the traversal. The same distance metrics was used also with the control group (Group B) whose system was based on the k-means algorithm. An example of the rendered Neural N-Tree in the students’ interface can be seen in Fig. 4. The users were able to open any of the nodes in the visualisation to expose the values assigned to the particular cluster (Fig. 5).

Fig. 5

Neural N-Tree rendered in real-time by the interface

The Group B used an EDM interface that allowed them to cluster datasets with the k-means algorithm. The interface allowed the students to change the number of the clusters, but otherwise the system worked in a black box, and the students did not have a possibility to change any other parameters or the model itself (such as the centroids). After the clustering process, the interface built result tables for each cluster. The tables show the dataset vectors in a cluster, and all features of the assigned vectors were visible in the table cells. An example of the clustering result in the tables is illustrated in Fig. 6.

Fig. 6

An open node from Neural N-Tree in the interface

The students were able to change the number of the clusters at any time, and they were informed that they should try to achieve as good silhouette index as possible. In addition to changing the number of the clusters, the interface for Group A students allowed adjusting the parameters and the node points in the AUI driven EDM system. The clustering outputs of the algorithms and silhouette index of clusters were recorder on each iteration that the students took to measure and compare how the clusters get developed during the iterations (Fig. 7).

Fig. 7

Clustering results of the k-means interface

The students were given 30 min to get familiar with the dataset, and try to record as high silhouette index as possible. After the students had completed the assigned task, they were asked to fill in a questionnaire, which was designed to measure the participants’ understanding about the clustering algorithms, and also their perceptions about the AUI method.

The questionnaire consisted of two parts. In the first part, the students gave their answers to the following open questions that were developed to capture the students’ understanding of clustering as a machine learning approach in general, and their comprehension about the experiment in particular. The questions tested the students perception about the cluster analysis process and how the cluster analysis process and the results helped them to understand the dataset.

• 1 What does the cluster analysis mean?

• 2 What the dataset you used was about?

• 3 How did the cluster analysis process help you to understand the dataset?

• 4 How did the final clusters help you to understand the dataset?

• 5 How did the final clusters help you to understand the differences within the dataset?

• 6 How would you describe the final clusters?

• 7 How would you describe the cluster analysis algorithm you used?

• 8 What did you learn about the context of the dataset that you clustered?

In the second part of the questionnaire, the students answered to 10 claims, that were measured with Likert scale 1-5. The questions measured whether it was easy for the students to conduct knowledge discovery with the algorithm and the EDM system they used. The questions measured also whether the cluster analysis process and the final clusters were easy to understand for the students.

• 1 The cluster analysis algorithm helped me to perceive the cluster analysis process.

• 2 The cluster analysis algorithm helped me to understand the dataset.

• 3 The cluster analysis algorithm helped me to discover new knowledge from the dataset.

• 4 The final clusters were justified.

• 5 I understand what cluster analysis means.

• 6 I understand differences within the dataset vectors.

• 7 It is difficult to understand why the dataset was clustered as in the final result.

• 8 It is difficult to discover new knowledge from the final clusters with the cluster analysis algorithm.

• 9 I learned something new about the context of the dataset with the cluster analysis tool.

After the both groups had answered to the questionnaires and used the EDM tools, we also ran tests with a state-of-art data mining tool RapidMiner with k-means and hierarchical clustering. The purpose of the additional study was to get even deeper understanding how the silhouette indices were set by different tools. We acknowledge that the data is not sufficient to compare the exact accuracy of the tools. Instead, the k-means implementation of the black boxed tool, the k-means and hierarchical clustering implementation of RapidMiner were used to see whether the performance level of the different tools and the algorithms would fit in the same range of accuracy in the given context. This also serves as a background information in the future studies where Neural N-Tree is used.

Algorithm performance

The analysis shows that both groups used their algorithms with varying number of final clusters (that is, the students experimented how change in number of final clusters affect to the final clustering and silhouette indexes). Five comparable categories were found from the set of experimented clusters. The highest recorded silhouette indexes for both algorithms are reported in Table 1.

Silhouette index is a measurement technique that shows how similar a data item is to its own cluster compared to other clusters. The silhouette index is a decimal value ranging from -1.00 to 1.00. Low positive and negative values mean that data might not be clustered to the correct clusters, and high positive values close to 1.00 mean that the dataset is clustered consistently. Because the students in Group A were able to adjust also the node points of Neural N-Tree, the initial silhouette indexes without adjusting the points were also recorded. In this way, it was possible to find out the effect point adjusting had in the process.

The results show that for every recorded cluster size, the silhouette indices of Neural N-Tree were greater when the point values of the nodes were adjusted. These results indicate that when the end-users adjust the models generated by Neural N-Tree, the accuracy of the model increases. Also, the widely used algorithms k-means for black box implementation of the control group (Group B) and k-means and hierarchical clustering for RapidMiner produced the silhouette indices quite comparable to the Neural N-Tree. The only significant difference can be found in the 2 clusters category. For 2 clusters, RapidMiner’s hierarchical clustering seemed to perform poorly. Besides that, the silhouette indices were not significantly higher for any of the tools in comparison to others. This indicates that the different algorithms performed quite equally for the dataset that was used in the study. This somewhat equal performance level of the different methods sets the ground for our assumption that the AUI method and Neural N-Tree would bring additional benefits to teaching and learning process by enabling more deep knowledge discovery process.

Survey about students’ perceptions

After the clustering task, the students were asked to fill in a questionnaire consisting of two parts. The answers were used to triangulate and deepen students’ answers in the second part of the questionnaire, which was a survey with 10 questions with Likert scale answer options (1-5; strong disagree - strong agree). We analysed the answers of the students by using Mann-Whitney U test for the given answers. The answers of the Group A were combined for each question and the answers of the Group B were combined for each questions. The combined answers were compared for the U value to validate whether there exists any significant difference for p<0.05 between the groups.

The results in the Table 2 show that there is no significant difference between the answers of the Group A and the Group B for any of the questions in the questionnaire (Table 3). However, as the descriptive statistics in Table 4 indicate, the variance of the answers for the questions was greater for the Group A (Neural N-Tree); the answers contained more scores from the extreme ends of the scale 1 - 5. Table 4 shows that both means and standard deviation between the two groups vary. In general, the mean values for the black box group (Group B) are higher than in the white box group (Group A). However, due to small sample size, it is difficult to make any final conclusion from these statistics. For example, the students previous experience in machine learning or related subjects may be reflected strongly in the answers. The results indicate that the from the point of view of knowledge discovery, understanding the dataset and the domain where the data was collected, the two approaches are quite equal.

Table 2 Recorded silhouette indexes

Table 3 U values and z-scores of the answers for the survey (scale 1-5)

Table 4 Descriptive statistics of the questionnaire (scale 1-5)

The second part of the questionnaire had eight questions concerning the students’ understanding about the clustering methods, as well as about general perception of the task and the EDM system they were using. One of arising themes from the answers was that students from Group A (Neural N-Tree) felt that 30 min was too short time to get familiar with the EDM system, and to comprehend how the algorithm works. Hence, their focus was clearly on increasing the silhouette index and not to understand the dataset, whereas Group B students indicated that they understood dataset and final clusters within the available time span. From this observation, we can conclude that achieving a good result when utilising a white-boxed EDM system and AUI method takes longer in terms of active working and exploration time than with black-boxed systems. One of the factors affecting to this might be the additional overhead that a more complex user interface causes. However, this issue and factors behind it need to be studied more deeply.

Students’ previous knowledge on data mining and machine learning was also reflected in the answers. Those who had been learning the topics during their earlier studies, had more positive perception for the Neural N-Tree EDM system and AUI method than those who were completely novices in the domain. It is important to notice, however, that all the students were computer science majors and in general, they are assumed to have a good understanding in programming and related topics. The focus of this study was not to assess learnability of the EDM system driven with AUI method, but the role of computing expertise is something to take into consideration during future studies when exploring the feasibility of the AUI method with end-users who may come from the different backgrounds to exploit EDM and AUI method.

Research stage 2

The teacher interviews

Two secondary school teachers used Neural N-Tree and AUI during and after the teaching sessions. The data that was collected for the analysis contained items from the students’ making activities in the classroom settings where the teachers were teaching. The making activities consisted of Arduino based crafting, design, programming, and eventually sharing the works among the other students in the class oom. The students used a special purpose web-based platform for the making activities and the platform passed data vectors to the EDM system that implemented Neural N-Tree. The data vectors contained information from the activities in the web-based making platform such as clicks of UI elements launching and closing different tools and state of the program for the Arduinos.

The teachers were able to adjust the generated models of NNT in similar way as the Group A students did in the first phase of the research. However, the aim of the adjusting and re-creating the models was not to increase the accuracy of the models,but instead to engage the teachers in the knowledge discovery process and facilitate understanding about the context where the data was collected.

After the making activity sessions in the classroom, the participating teachers were interviewed. The teachers were asked to describe their usage of the NNT implementation and AUI method to reflect their experiences. The main interest during the interviews was to evaluate the level of knowledge discovery that was achieved during the use process, to understand how easy it was to use the EDM system, and finally to evaluate whether the teachers would prefer the AUI method over other EDM methods. The interviews were led by the researcher. Both of the interviews were recorded and the recordings were compared to each other to find congruent discussion. Some of the themes such as trustworthiness and the level of knowledge discovery were recurring and thus, taken into account during the analysis. The interviews included free discussion and the following questions that were asked from of the teachers.

• 1 How do you find EDM in general?

• 2 A performance of a predictive algorithm can be evaluated. In the context of EDM this could mean that the used algorithm predicts a grade of a student accurately. Which one do you prefer: the prediction is correct or that the use of an algorithm leads to a knowledge discovery?

• 3 In the terms of knowledge discovery, do you think that AUI method can lead to it?

• 4 In the AUI, you were able to adjust the predictive models through the visualizations and thus you had a full control of the analysis process. Which one do you prefer: an open approach like this or a black-box approach were the user has little or no control in the analysis process?

• 5 Did you find the use of AUI challenging or time consuming? Why?

• 6 Which one do you prefer? A machine would generate the predictions or you and the machine would generate the understanding in the educational context together?

For the first two questions, the teachers regarded the EDM to be a supportive tool that would deepen the domain knowledge for a certain context. That is, the teachers preferred knowledge discovery over the accuracy. The answers to the third question pointed that it is indeed possible to generate knowledge through AUI (and NNT); the combination of the visualization and the adjusting process seemingly increases the understanding by combining the perception of the algorithm on the dataset and the domain knowledge of a teacher. Whereas the teacher posses the intuition and the experience, the machine tends to produce the generalization from the data.

About knowledge discovery

Can lead to knowledge discovery. Because the rules are visible. Visualized the differences between different level of users.

The fourth question compared the traditional black box approach and the AUI method in EDM. Neither one of the approaches was viewed superior to another. A learning curve and time consuming nature of the AUI was seen as negative characteristics. Although, the other teacher noted that if the system with a AUI method would be more automated and easier to use, then the approach would be favourable.

The fifth questions asked whether it is challenging or time consuming to use the AUI. Both of the teachers considered the system to have a steep learning curve. There seems to be too many options and possibilities to consider and too many technical terms, which require previous knowledge in machine learning. However, the visualizations of the generated Neural N-Tree models were seen easy to interpret. One of the two teachers pointed that there would be more motivation to study the system if the projects would have last longer period of time. Within the current time frame, there was only a little bit motivation to study the system. In the case of the sixth question, both of the teachers favoured the idea of AUI. The process of AUI should, however, be easy to learn and automated to some extent. The other teachers mentioned about the learning curve :

Even though I understood the tree model (NNT), it took me a lot of time to really understand it... For small projects, learning it would not be feasible.

The last questions was about the improvements that could be implemented in the system. The challenges that emerged during the previous questions were also present in the answers for the last questions. The system was seen challenging and because the making activities that the teachers taught in the classroom only lasted for a month, there was no or little motivation to study the system. Based on the answers, the AUI method itself was seen positively but the usability of the system could be improved.

During the free discussion, both of the teachers noted that when the predictive model is visualized and adjustable, it is easier to trust the analysis because the clustering process is visible for the end-user. Furthermore, if an algorithm produces a Neural N-Tree model, which in the teacher’s point of view does not generalize the features that have an influence in the similarities of the students, the teacher is able to adjust the model accordingly. The other teacher noted that

The machine justifies the decisions. In my opinion, because of the justification for some tasks is easy to trust.

The teacher questionnaires

The workshop for the teachers was organized in the fall of 2018. Around 30 teachers participated in total. The participants used a web-based platform for making projects. The teachers worked in small groups of 3 to 7 people. Each group chose a theme for the project and the actions within the web-based making platform were collected to be analyzed in the system that implemented Neural N-Tree algorithm. Each group selected one teacher for the EDM task. Other members of the group were programming, planning and crafting the artefact. The selected group member analyzed the data collected from the learning process. For the analyzer, we showed how the use the EDM system and also gave a brief introduction to the educational data mining. In total, the making activities lasted for 4 h.

After the use of the AUI driven data mining tool, the analyzers were given a questionnaire with 9 questions that were related to the educational data mining (EDM), the knowledge discovery levels of NNT and AUI and the user experience. The questionnaire used Likert scale where the teachers chose the most describing option for the questions from 1 (completely disagree) to 5 (completely agree). The questions presented to the teachers were

• 1 I understand what Educational data mining means.

• 2 I understand how to use the platform that implements AUI.

• 3 I would be able to analyze my students with the platform that implements AUI.

• 4 I can trust the analysis when the predictive model is visualized for me.

• 5 I gain new knowledge when I see the models visualized for me and I can change them.

• 6 AUI method is easy to use.

• 7 Knowledge discovery is more valuable than correct predictions in educational data mining.

• 8 I understand why visualizations and possibility to adjust models can lead to new knowledge.

• 9 I would use NNT and AUI with my students.

The questions were designed so that the answers would be related to the previously conducted interviews with the teachers. The hypothesis for the questionnaire was that the answers given by the teachers for the questionnaire would support the answers given by the teachers for the interviews. Table 5 shows the answers of the teachers for the questionnaires as well as the mean and SD for a specific question.

Table 5 Teachers’ answers to the questionnaire

The results from the questionnaire support the previously conducted researches with the computer science majors (research phase 1) and the interviews with the 2 teachers (research phase 2). AUI method was not seen easy to use in general (Q6). However, the question about the knowledge discovery (Q5) had the lowest SD and the highest mean among the questions (in addition to Q1 that asked about whether the teachers understand what EDM is). This indicates that AUI (and Neural N-Tree) indeed facilitates knowledge discovery. During the interviews, the teachers noted that it is actually easier to trust the analysis when the model is adjustable and visible for the end-users. In the questionnaires, Q4 had the highest SD among the answers, which indicates that for some teachers it is easier to trust the predictive model and the analysis while some teachers disagree.

In general, the answers to the questionnaires and the interviews are congruent. AUI requires time to learn and has a steep learning curve. However, the end-users seem to agree that the use of AUI (and Neural N-Tree) helps in knowledge discovery.

Limitations of the research

The limitations of the research are related to the small sample sizes. The results indicate that the performance of Neural N-Tree algorithm increases over time when the models are adjusted by the possible EDM end-users. However, in the first phase we only used one dataset, which consisted of exam scores and hence, the dataset itself was easy to comprehend. The conclusions on the accuracy improvements in Neural N-Tree with AUI method require is limited to the context of the study, and more general conclusions require more research.

The interviews and the teachers’ answers to the questionnaires indicate that the knowledge discovery is possible with AUI. In general, all of the answers were in line with each others and the teachers agreed that AUI leads to knowledge discovery. The generalisability again suffers from the small sample size and the the teachers used AUI and Neural N-Tree only in one context that was familiar for them.

The first part of the study with the computer science majors also indicated that the AUI method leads to knowledge discovery. However, the students who took part to the research were not familiar with the context where the dataset was collected. We introduced the dataset and the features but it is possible that the true knowledge discovery in AUI method occurs when the end-users are familiar with the context. Otherwise the subjective experience and new knowledge rising though AUI method might be biased that does not reflect the factual knowledge of the educational context.

In the future, AUI and Neural N-Tree need to be tested with more potential EDM end-users and multiple datasets to more deeply evaluate the accuracy of the algorithm and the adjustable models.

Conclusions

Based on the previous research (Jormanainen and Sutinen 2013;Delibasic et al. 2013;Kovanović et al. 2017;Zhang and Qin 2018;Slater et al. 2017;Asif et al. 2017;An et al. 2017;Fernández and Luján-Mora 2017;Khan et al. 2019), we made an assumption that end-users benefit from the open-ended EDM process in the form of more interpretable and contextual support and decision making tools. We have presented in this paper a study that was conducted to asses whether the AUI method and the implemented Neural N-Tree would increase its accuracy over time when the end-users were adjusting the predictive models.

We have introduced AUI method for EDM, where the understanding about the educational context is created in a collaboration between a computer and an end-user. In the AUI method, learning is iterative and cumulative; the computer builds a model from which the end-user discovers knowledge and adjusts the created model based on the subjective interpretation. The computer, in turn, adjusts the model based on the interpretations that the end-user has expressed in the system.

Our results show that an EDM system following the AUI method can produce more accurate results over time but there is was no significant difference between of the different algorithms in the terms of silhouette indices. Also, using only one dataset and one metrics for the cluster evaluation is not sufficient to show that one algorithm would be superior to another. Our results show that achieving a good result with the AUI method takes longer time and more iterations than with the black boxed algorithm. Understanding the approach and the associated tools takes longer time, but despite the longer process, the AUI method helps the EDM end-user to discover new knowledge from the context more easily.

Furthermore, the results indicate that those who have previous knowledge about data mining related topics, were more comfortable using the AUI method than those novices in the context. The test participants from the first stage of our research with no previous experience found it challenging to understand mechanism behind Neural N-Tree and concentrated more in increasing the silhouette index. These findings suggest that the AUI method is beneficially especially to mediocre users with some degree of existing knowledge about educational data mining and learning analytics systems. However, the Likert scale for the questionnaire showed no significant difference between the answers of the groups using either k-means or Neural N-Tree when for instance the level of knowledge discovery is being measured.

The second stage of the research casts the light to the knowledge discovery process of AUI. The results from the second stage of the research indicates that AUI method leads to knowledge discovery but is not easy to use. The participants implied that the knowledge discovery is possible but unlike in the first phase of the research, the test subjects were also familiar with the dataset context and the dataset. Also, some indicated that when the predictive model (in this case the Neural N-Tree) is visualized to the end-user and is adjustable, it is easier to trust the output of the model. However, the trustworthiness of AUI depends on the user and not all of the test subjects in the second stage of the research found AUI to be more trustworthy.

In this paper, we introduced two research questions, for which we have provided the answers based on the conducted research. The research questions were as follows:

• 1 Does the accuracy of a predictive model increase over time through Augmented Intelligence method?

  The resulting silhouette indices indicate that the Augmented Intelligence method and model adjusting increase the accuracy, hence leading to better context-specific models.

• 2 Are there any indicators that the Augmented Intelligence method leads to the knowledge discovery?

  The results from the study indicate that the knowledge discovery is possible though the Augmented Intelligence method especially when the context of the dataset is familiar to the EDM end-users.

The results indicate that AUI method in an EDM system is challenging for the end-users who do not posses of required skill set in data mining. On the other hand, the results indicate also that especially mediocre users may benefit from the AUI method when they are able to use enough time to comprehend the system and focus on knowledge discovery process through the AUI method.

Future work and discussion

The results in this paper indicate future directions for the research. We will experiment the AUI method in real educational contexts with bigger end-user groups. Also, one of the aspects for the future development include expanding the selection of the algorithms in the AUI method with decision tree classifiers. Decision trees have proven to be easily interpretable even by non-expert teachers (Jormanainen and Sutinen 2013). In such systems, a decision tree is rendered in the system and the teacher can modify the nodes in the visualisation and in this way have an influence in the model produced by the algorithm.

In the future studies about AUI method, the longer time requirement for familiarising with the system has be taken into the account. For computer science majors, a binary tree and the traversals in the binary tree are quite easy to explain when comparing to non-expert end-users such as the teachers, who have no experience in data structures or data mining whatsoever. This sets requirements for sufficient introductory material to scaffold the novice users (teachers) when they are using the system.

The model of the AUI method can be seen as a product of both end-user’s and computer’s interpretation of an educational context. Changes to the model made by the end-user always reflect to the subjective experiences and knowledge of the user. The subjective understanding from the context forms a basis to the knowledge that the computer and the user generate. In the field of computer-based modelling, Empirical Modelling (EM) (Beynon 1997) have a similar relationship between the model - the construal - and the phenomenon being modelled - the referent - and it is called state-as-experienced (Harfield 2007). This emphasises that a state of a construal can not in general be viewed in the way that a state of a traditional computer program is viewed, but a construal is far more subjective. The AUI method possess similar view as the state of a model produced by the AUI method is always subjective.

Because the AUI produces models, which are described by the subjective understanding of a end-user, the future evaluations of the AUI need to take the subjective nature of the model and the results into the account. For instance, a question arises whether a silhouette index is a good metrics for subjective knowledge in educational settings? Also, knowledge generated in collaboration between a computer and an end-user does not solely depend on the predictive results of the approach but also the knowledge is implicitly generated by the collaboration and knowledge sharing.

Furthermore, the visualisation of a predictive model, whether the model is for clustering or for classification, plays a crucial role in in data mining these days. Such visualisations can be made if the model is relatively small. For instance, if a decision tree classifier would be used as an algorithm in the AUI process, the interpretability of the visualisation suffers if the depth of the tree is too high. Thus, AUI should be used for relatively small datasets where some of the commonly used methods, such as deep learning algorithms, fail to build the generalisations. This sets also a requirement to develop novel computational methods specifically for the AUI method.