Table 1 Example data extraction and analytics matrix

Cognitive

Problem decomposition

Identifying a logic of initial programming codes

• Logic identification

 ◦ Programming code tagging (v)

 ◦ Verbal annotation records on each code (v)

 ◦The amounts of created codes (n)

 ◦Document-similarity score (n)

• Verbal reflection records: Topic modeling (Latent Dirichlet allocation: LDA)

• Normalization of keywords: bag-of-words

• Document-similarity analysis: TF-IDF (cosine and Jaccard indices)

• Extractions of key coding patterns: SPM (GSP and SPADE algorithms)

• Score normalization for data fusion: CombSum and ComMNZ

Abstraction

Required programming code formulation

Fluent representations of code formulations

• Code formulation

 ◦ Programming code tagging (v)

 ◦ Total number of codes (n)

 ◦ Frequency of key coding patterns (n)

• Fluent representation

 ◦ Total number of new codes (n)

 ◦ Procedural code similarity

Automation design and development

Developing a required loop system

Building an efficient loop structure

• Loop-system development

 ◦ Frequency of key coding patterns (n) – specific to the loop system.

 ◦ Correct expressions of loop codes (v)

• Loop-design efficiency

 ◦ The number of codes (n)

 ◦ Required implementations of loop codes (v)

 ◦ Total time to modify codes on an application (n)

Testing and Debugging

Effective modifications of existing code structures

Identifying and correcting error codes

• Effective modification of existing code structures

 ◦ Total number of runs (n)

 ◦ Frequency of changes in existing codes (n)

 ◦ Total time to modify codes on an application (n)

 ◦ Verbal annotation records on each code (v).

• Identifying and correcting error codes

 ◦ Frequency of correct and erroneous runs (n)

 ◦ Total number (time) of failures on runs (n)

Cognitive load

Mental exertions in programming

• Mental effort estimation

 ◦ Code development efficiency (n)

• Efficiency estimation (the number of codes / (Total time for code developments) with z-score normalization

Prior knowledge

Learners’ declarative and procedural knowledge as to programming languages

• Code formulations by using pseudo-codes

 ◦ Verbal annotation records (v)

 ◦ Frequency of erroneous implementations (n).

 ◦ Attemps to change values on pseudo codes.

Emotional

Plutchik’s six basic emotion (e.g., sadness, surprise, trust, and joy)

Articulations of emotional expressions

Behaviors that represent positive and negative emotional states

• Engaged states to participate in tasks (positive).

• Quitting a system (negative)

• Articulations of emotional expressions

 ◦ Real-time chatting logs (v)

• Engaged states to participate in tasks

 ◦ Pause time (n)

 ◦ Frequency of starting a new task (n)

• Quitting a system

 ◦ Frequency of attempts to quit a system (n)

• Emotional state classifier (probability estimation; i.e. emotional predictor designed by recurrent neural network)

• Topic modeling (LDA)