Effectiveness of peer review as cooperative web-based learning method applied out-of-class in a role playing game: A case study by quasi-experimental approach

Abstract

Education at universities requires a high proportion of self-study supported by instruments which motivate students to deepen their knowledge with case studies and training software. The gain in learning is higher when what is learned is reflected by others in a cooperative process. The Class Peer Review (CPR) requires from students to evaluate each other but lecturers lose a lot of time when applying it during the lecture. Research on CPR shows many different approaches to how this method should be carried out effectively and many of them still demand a high presence of the lecturer. In contrast to most studies, this article reports the results of a CPR conducted outside the classroom and assisted by a web-supported role playing game representing a publisher where a common online class journal is used as an incentive and motivational element. The effectiveness of CPR was examined by means of a quasi-experimental study. It turned out that the experimental group used the training software significantly more frequently in addition to the lecture materials and a meaningful increase in group dynamics was recorded. Also female students were more motivated to continue this type of group work in the future. Final grades were better under CPR conditions, but the effect was statistically weak. Basically, the CPR method seems to be an effective tool to extend problem-based teaching to the self-learning phase if practiced in combinaktion with a role play and fictive Class-Journals as incentive. This provides further potential to e-learning instruments.

Highlights

Effectiveness of peer review as cooperative-learning method applied out-of-class

• Studying at dual Universities demands high self-responsible learning

• Class-Peer Review (CPR) applied out-of-class was tested for effectiveness

• Group-dynamics was improved and findings are partly gender-specific

• The effect on final class-grades was low but more pronounced for practical topics

• CPR effectively combined class teaching with self-study phase

Introduction

Problem description and research questions

The engineering studies at universities are differently focused on practical applications. It ranges from predominantly method-theoretical subjects to a dual approach. In the latter case, students usually spend up to half of their studies in industry in a training workplace. The dual study is primarily a German model that emerged in the 1980s. Due to its success and in particular promoted by German companies with foreign branches, it has already been established in neighbouring countries such as France, Spain and Netherlands (Bode et al. 2012). Since a few years the transfer and the integration into the local educational structure of further countries like Mexico, USA, Qatar and Brazil is being prepared (Graf et al. 2014). In the conventional lectures at universities, students take part in courses for approx. 16 weeks that follow a fixed schedule. Several subjects are arranged sequentially throughout the day and are repeated weekly. Participation is often not compulsory, but recommended for passing the obligatory course assessment. The lecture period is followed by approx. 4 weeks of exam preparation and execution and finally by 6 weeks of semester holidays. With the dual study concept, on the other hand, students spend half of the 6-month semester at the university in lectures and in a training company with which they had previously concluded an employment contract for a three-year Bachelor’s programme. The lectures run for 13 weeks with immediate final exams. Then the 14-week practical phase follows. The class times are obligatory for students and mostly divided into frontal teaching of in total 180 lecture hours plus 360 h self-study time at home. The latter require a high degree of self-motivation. In addition, the individual lectures are irregularly distributed throughout the semester, as many lecturers come from industry and thus from outside the university and they schedule their frontal teaching compactly, i.e. for 4 to 5 h each. So there are mostly only two courses per day to attend, but each one lasts very long and there are no semester holidays at all. This requires even more a high learning discipline and willingness (Baden-Württemberg International 2017). However, the relevance of the lecturers towards the industry allows the greatest possible orientation of the teaching and learning material to the engineering practice and often includes the application of specific software. This makes the students familiar with the work in design offices. However, the demonstration of a software application in class teaching takes a lot of time. Some students can only slowly follow the steps of the lecturer on their own computer. If they made wrong inputs they easily lose connection to the lecture. This could have a negative impact on the grade in a written final examination, which is typically held in technical courses. Taking more time for in-class software training has the danger that the software demonstration itself will become a frontal lecture where independent thinking and trying out will be neglected. However, the high percentage of self-study time in such dual study programs offers ideal opportunities for students to familiarize themselves with special engineering software by working independently in software labs (DHBW 2019). In doing so, they gain valuable experience knowledge close to the practical cases from their training companies which provides them in particular with analytical problem-solving competencies. But this requires guidance from the teacher.

The aim of this study is to find out whether the Class Peer Review (CPR) method represents an effective means of indirect teacher influence where the latter only controls the process and the classmates teach each other in a web-based virtual environment outside the classroom. In particular, it is of interest if the learning outcome in a final written exam actually benefits from CPR. The study applies a quasi-experimental research design where two cohorts were selected as courses on “Building Physics” in higher education, predominately attended by men, and instructed by the same teacher in the third semester. A high proportion of software exercises were consciously embedded. The experimental group was taught 48 h conventionally in the classroom and then had to conduct a Class Peer Review within the self-study phase with a total of 52 h, hence out of class. In the same period, the comparison group received only the instructional and frontal teaching from the same lecturer and could freely decide on their 52 h of self-study time. Both cohorts thus received an equal teaching content and only the CPR made the distinction. The study tried to answer the following questions using ANOVA descriptive statistics to find out about differences between or within the two cohorts and the effect size regarding the following specific variables:

(1) Does CPR applied outside the class in a role play gaming situation make students working more independently with the software? (Sect. 3.2.1, Variable (1) “Acceptance of Learning Resources” - between cohort analysis); (2) Does CPR applied outside the class contribute to a greater feeling of togetherness throughout the course? (Sect. 3.2.3, Variable (4) “Group Dynamics” - between cohort analysis); (3) Does CPR applied outside the class provide comparatively more motivation to women in a technical discipline like “Building Physics”? (Sect. 3.2.3, Variable (4) “Group Dynamics” - within cohort analysis); (4) Is CPR well accepted by students and therefore effectively complements a direct teaching environment? (Sect. 3.2.5, Variable (6) “Acceptance of the Teaching Method” - between cohort analysis) and (5) Does CPR have a positive effect on the students` final scores? (Sect. 3.4, Variable (7) “Effect of CPR on the Lecture Exam” - between cohort analysis).

Problem-based learning (PBL)

Research on teaching and learning theory examines the transfer process of information from lecturer to learner as well as the acquisition and further processing of the information itself by the recipient. Theories put individual aspects of learning in the foreground and try to explain which prerequisites make learning particularly effective. The Constructivist Theory sees learning as an active and self-directed process of the learner. The learning success itself depends on the context in which the information is placed. For example, information is best incorporated if it interacts with existing know-how and constructively develops one’s own world of knowledge. Other theories see the social learning environment as a motor for successful study. All the more so when like-minded people take part in the learning process and even actively shape it. By 1976, Bandura had already put forward the thesis that participation in groups has an extrinsic effect on the learner. According to the Social Interdependence-Theory, learning success very much is determined by how supportive other learners are in knowledge transfer. Piaget (1965) and Karami and Rezaei (2015) see a stimulating component in this context, especially for cognitive activities in the learner but also in the learner’s partner. This stems from the fact that both participants exchange arguments and concepts and evaluate the information more critically than is the case with unilateral knowledge transfer from lecturer to student. Johnson and Johnson (1994) see reinforcing effects in learning when a “we” feeling is implemented among learners under common targets with the awareness that they can also fail together.

Obviously, Cooperative-Learning (CL) has a reinforcing effect when embedded in a conventional instructional learning environment. A fundamental orientation of teaching towards the principles of the PBL is given when students are offered the opportunity in the lecture to test their theoretical and instructional knowledge in practice-relevant situations. This can be done in case studies where the application of a software, to solve specific engineering problems, play a main role. In addition, the learning effect is enhanced when the study environment resembles a role play (Erturk 2015). Moreover, Gallagher et al. (1995) see a greater success in learning if students use their own data and concepts more intensively and are finally allowed to present their solutions to others. Stark et al. (2010) found that a sequential order of instructional and problem-based teaching units in semester lectures is less effective than a purely BPL-based learning environment. The latter, however, needs guidance and control mechanisms during implementation, which requires a comparably high attendance of the lecturer. In this respect, a web-based environment can save a lot of work (Sun et al. 2019). Further, it can even turn CPR into a role play to enhance its impact. If the mutual review-process indeed is reinforcing the learning progress was investigates by Papinczak et al. (2007). The authors tried to find out if a peer assessment conducted in-class had a positive effect on the self-perception and engagement of medical students within a PBL-teaching unit. In small groups, they analysed practice-relevant cases of diseases and gave each other feedback on their solution concepts. The learning advantages were seen as being significant which is why the authors suggest conducting further research on CPR’s integration in predominately PBL-oriented lectures.

Class peer review (CPR)

Class Peer Review is a method by which students apply their instructionally acquired knowledge in an independently conducted study on a practical topic (Nortcliffe 2015). Subsequently, the execution and results are reported in a manuscript (Sims 1989). What is special about CPR is that at least one fellow student revises the manuscript, gives critical feedback and the author is finally asked to improve the manuscript accordingly (Xiong et al. 2010). Tighe-Mooney et al. (2016) describe this type of learning and teaching process as peer tutoring, peer instruction or peer assisted learning. Most studies on CPR report about improvements of individual study results. Sims (1989) demonstrated a significantly higher quality of the term paper which was attributed to the constructive criticism of fellow students. However, whether the authors had already produced a better term paper by the announcement of the review alone remained open. Stark et al. (2010) went a step further here and examined the factors of autonomy experience and social involvement and were able to prove that these were also improved by PBL. Further factors, such as differences between female and male participants or whether CPR can be used specifically for higher acceptance and use of certain teaching materials, e.g. a software, have not yet been investigated in more detail. However, Cho and Schunn (2007) stated that CPR is more effective in groups which are very heterogeneous with regard to the chosen research topic and writing skills. In order to achieve the latter, Sun et al. (2019) designed an algorithm which, according to the greatest Euclidean distances between students over their past academic achievements, assigned a strong reviewer to each weak author. In general, under CPR all participants learn particularly from the process of data analysis, evaluation of results, presentation and explanation of the findings and the revision of their own work in accordance with the review report (van den Berg et al. 2006). This shows that CPR has a high potential to integrate web-supported tools. Current research describes the use of new e-learning systems such as SWoRD (Leijen 2017). Also in use are conventional Moodle-platforms that help guide the review process and can be extended with specific plug-ins (Sun et al. 2019). However, these only optimise the CPR process and aim to improve students’ writing skills. Therefore they are rather impersonal and neutral from the user interface. It puts students only partially in a realistic role as authors. A web application should therefore also help to discuss and present the subject-related contents of the manuscripts in a suitable setting within the role play.

Some studies on the effectiveness of CPR refer to technically oriented disciplines. Zejnilagić-Hajrić et al. (2015) practiced PBL in combination with CPR in chemistry laboratory with practical exercises and found that CPR is superior to the teacher-centered learning environment. Moore and Teathe (2013) used CPR in mathematics courses, Kelly (2015) in a biology laboratory lecture, Leijen (2017) in a chemistry class and Sun et al. (2019) tested it in a course on computer language programming. All studies show that the autonomy experience and the group dynamics are positively influenced. Other research focuses on science and language courses. E.g. Shirgurkar (2018) conducted a study in a foreign language course whereby Lyman and Keyes (2019), Aydawati et al. (2018) and Gordijn et al. (2018) investigated the improvements in students` ability to write a research proposal. Not all of these studies use questionnaire-based pre- and post-surveys to quantify the effectiveness, and also the impact on the final exam grade itself is rarely investigated. This is most likely because courses under comparison were taught successively or the lecture did not require an objectively assessable written exam. This, on the contrary, was the aim of a study by Celik et al. (2011) in a physics lecture and by Crowe et al. (2015) in a course on research methodology. In the first case, student results were significantly better in the final exam and in the latter case, no significantly different results could be observed compared to the other group. Whether CRP is even more successful in engineering courses cannot be proven yet. In all these examples, CPR was practised in-class, i.e. during the attendance lecture. In this context, Crowe et al. (2015) suspect that out-of-class peer review can be more effective. The reason is that the participants could become more flexible and motivated in conducting the review and the revision, which might lead to a stronger learning effect. In this regard, Lyman and Keyes (2019) recommend CPR as the best way to reduce students’ fear of wrong performance which should lead to better results when the teacher is absent.

This study is intended to supplement the research results to date, which have so far mainly focused on in-class applications and differ much in learning content and conditions. The peer review was therefore conducted at a Dual University and exclusively during the students’ self-study hours over several weeks according to a strict timetable with deadlines. A web-based environment was created to support the role play between the publishing house, which is the lecturer, and the students as authors. In order to maximize the self-motivation of the participants, it was announced to publish a fictive Class-Journal on the web-platform containing all contributions, so that everyone from the course could compare their performance with that of their fellow students. At the same time it is an incentive as it reveals insufficient engagements. Also in contrast to most papers, this study examines the impact of CPR on student performance in a final written exam as the main assessment method in the teaching of technical subjects. Similarly to previous studies, qualitative aspects, such as student motivation and group attitudes, were measured in order to better estimate the different forces of effect within CPR.

Research method

Study design, variables and instruments

The quasi-experimental study between an experimental (n = 31) and a control group (n = 29) followed a pre- and post-test design to collect data from a lecture held at two study courses. A similar case study was conducted by Rouhi and Azizian (2013), Zejnilagić-Hajrić et al. (2015) and Aziz and Hossain (2010). The execution was at the Cooperative State University of Mosbach, Germany, Department of Engineering, with students enrolled in Civil Engineering, 3rd semester in the course “Building Physics”. For both classes, the setting consisted of a conventional teacher-centered learning environment as a classroom with a video beamer, over which instructional teaching was carried out on the basis of presentation slides. A total of 48 h were held in 9 blocks of 5 h each and a 3 h closing session. As activating elements, the author first calculated 3 to 4 times per teaching sequence exercises on the blackboard and finished them in dialogue with the students and finally the students themselves in groups of two. The main component of the lecture is also the software application, by means of which the lecturer presented the solution of various practical issues. During the demonstration, the students had the opportunity to calculate on their own computer, which was not always the case. Nevertheless, the participants were able to use the software at home at any time alongside with lecture slides and sample solutions. It should be pointed out here that this lecture concept was absolutely identical for both groups.

In the first lecture the CPR concept was introduced to the experimental group only and its application and assessment was agreed upon. Thus, the comparison group was not informed about the CPR performed in the parallel course which took place at other times anyway. Both groups also came from different specialisations in the study programme, so that they had no regular contact with each other. Thus, self-selection could not occur. The equivalence of both cohorts was proven using questionnaire-based pre-tests (see Table 1). A reliability test for the questionnaire showed a high internal consistency (Cronbach’s Alpha = .90), which is why an inner validity and a high control of potential disturbing variables can be assumed. Evidence of similarity between the groups´ mean attitude was investigated by cross-case analyses using an independent sample t-test at 5% significance level which is similar to the studies carried out by Stark et al. (2010), Hossain and Tarmizi (2013) and Lyman and Keyes (2019). In the first part, the participants of both groups anonymously revealed their biographical data, such as age, gender and educational level. Table 1, upper part, compares the two groups in terms of the data collected. It is evident that in particular the biographic variables of both groups are not different, i.e. not statistically significant for p ≤ .05. The experimental group is a little younger and with 27% shows a lower proportion of women compared to the comparison group with 38%. A possible association of both groups was additionally excluded by Cramer-V and Chi²-test contrast studies. Both revealed no correlation between age and gender (experimental group: Chi²(1) = 1.55; p = .213 (n.s. = not significant); Cramers-V = .22 and comparison group: Chi²(2) = .13; p = .937 (n.s.); Cramers-V = .06).

Table 1 Pre-test variables and significance test between both cohorts

The second part dealt with their motivational attitudes towards the subject of “Building Physics” and software applications, which were clarified using a rating scale (Table 1; lower part). In contrast to Lyman and Keyes (2019), 6 points were used to ensure that the students clearly commit themselves to a positive or negative statement. In particular, their motivation to participate in this subject (Table 1, Variables 2.1 + 2.2 + 2.7), their opinion on collaborative learning (Variables 2.3 + 2.4) and their willingness to work with software (Variables 2.5 + 2.6) were of interest. It was found that only the experimental group points to a slightly higher motivation in this course subject, which nevertheless only reaches 65% of the maximum value. Weakly significant is that the experimental group comparatively prefers to learn alone instead of seeking support in the group. Overall, however, both groups appear to be homogeneous.

After the complete lecture of 48 h, hence prior to the exam, a post-test was conducted in both groups to reflect their learning process according to 6 motivational dimensions associated with their learning success. Its internal validity was also proven and the data were used for an ANOVA test conducted again at 5% level and for calculation of Cohens-d effect size. The correlation of the results between the two groups is graphically presented in Figs. 3, 4, 5, 6, 7, 8, 9 (Sect. 3.2) by the exposed mean item-scores and their standard deviations.

Implementation of the CPR method

In the first lecture unit, the experimental group received corresponding information about the objectives and the purpose of the CPR-procedure by means of slide presentation. They were also shown a detailed flowchart with deadlines for submitting the manuscripts, for providing the review and for resubmitting the final version of the manuscript. In addition, a so-called “Guide of Authors” was made accessible to all participants. This Word-template contained instructions and a sample article with the obligatory formatting. The manuscript had to be exactly 2 DIN A-4 pages long and the structure was given as an infobox with keywords, an abstract, chapters about introduction, methods, results, discussion of the results, summary and reference list. This corresponds to the usual structure of scientific publications in journals and was earlier similarly carried out by Sims (1989). The lecturer also created a cover page with a foreword for the In-Class Journal “Basics in Building Physics (JBBP)” which was meant to be an incentive for the class to actively contribute to the process. Together with all finished contributions of the students a final approximately 70-page journal resulted at the end and serving as exam preparatory material and learning resource for further studies. Since every article could later be accessed by class fellows, there was pressure to participate seriously in the journal. Similar to a CPR-research by Tighe-Mooney et al. (2016), there was a rubric as a pdf-document for the review. A part of it contained seven control questions on the formal structure of the manuscript, e.g. whether at least 5 references were given, and in the second part another 8 questions on content judgment, e.g. whether the problem is comprehensible or whether the results from the software calculation are correct. All these supporting documents were stored on a website created specifically for the review, to which the students were given access via a password and which was part of the role playing game. The lecturer was representing the editorial office and maintained a list of the deadlines and the names of the randomly selected reviewers during the procedure. Students were contacted by the editor at fixed dates via e-mail which was the same and impersonal to everyone. All participants had to submit their respective results online to the publisher. This came close to a game in which one should identify with the role and thus support the success of the CPR, while the person of the lecturer remained in the background.

The entire CPR took place parallel to the 10 classroom lectures which were held in weekly intervals. However, all students` activities relating from the CPR were conducted in their private self-study time whereas students from the control group could freely dispose of their spare time. In both classes the software was mainly discussed in the first three lectures. In the further course, the students of the experimental group were able to carry out their own study during the self-learning period after the lectures. This study should have the greatest possible relation to the main activity of their internship company. The use of the software was mandatory. As can be seen from Fig. 1, by the 4th lecture the students of this group had to prepare a manuscript about their study results according to the specifications in the Guide of Authors. The manuscript should be submitted to the publisher (=lecturer) by the 6th lecture at the latest. The latter then forwarded it to the reviewer as an anonymous manuscript in the so-called double-blind method. The reviewer had to examine it within 7 days according to the rubric. At the latest after the 7th meeting finally also the last review had to be completed, so that the lecturer had time up to the 9th session to create the journal from it. Afterwards it was presented to the course in the 10th and last lecture as a joint group achievement. As can be seen, the lessons for both groups took place exactly the same in sequence and content so that solely the CPR could make the difference between the groups. All communication between the editor and authors took place outside of presence time and exclusively by e-mail. Verbal instructions during the lectures were only given with regard to keeping up with deadlines and as a reminder to use the obligatory Word-template. The net-teaching time was therefore identical between the cohorts.

Fig. 1

Scheduling of the in-Class Peer Review procedure over 10 weeks for the experimental group

In a similar way study results were disseminated in a research by Cho and Schunn (2007). In contrast, at Shirgurkar (2018) the single results were only discussed in a plenum in the class in order to simultaneously gather opinions from others also. The methodology of Gordijn et al. (2018) differs in that each author could rebut a point of critique in the review and the reviewer had to comment on it again. The purpose of this is to iteratively rethink hasty criticisms, which also contributed to improving the CPR’s quality but was associated with considerably higher administrative effort. In the present study, the review was only carried out once. The fact that the impact strength might not be sufficient to positively influence the exam results was accepted in favour of a manageable effort. Nevertheless, this point is not negligible because a study by Leijen (2017) shows that authors are three times more willing to revise their manuscript according to the review if the critique was made by two reviewers independently of each other. Finally, it was important that the review was carried out solely by fellow students, in contrast to a study by Lyman and Keyes (2019), which also integrated feedback from the lecturer. In comparison to current research literature, the study presented here avoids additional quality-enhancing elements in the process and tries to create incentives via the journal only, which also represents a punishment by revealing any inactivity on the part of the student. This somehow corresponds to a recommendation by Sun et al. (2019), who postulate the CPR to become a dynamic game with reward and punishment.

Post-evaluation

In the last lecture, a post-evaluation was also carried out in both groups by means of a questionnaire using an 11-point rating scale according to Wagner et al. (2013). It contained questions on 6 different areas, which represented the dependent variables in the study design and were partly derived from Stark et al. (2010) and van den Berg et al. (2006). They cover the topics (1) Acceptance of Learning Resources as slides, scripts and software, (2) Autonomy Experience as a willingness to sacrifice private time, (3) Feeling of Success, (4) Group dynamics, (5) Experiencing Own Competence and (6) Acceptance of the Teaching Method. In order to clarify if women were comparatively more motivated by CPR, the students again had to state their gender. Similar to the pre-evaluation, the internal consistency of the follow-up assessment was high (Cronbach’s Alpha = .90). The 7th variable corresponded to the exam grade. This was divided into knowledge-based issues and transfer questions with practical relevance, where skills in the software are indispensable. Both areas of expertise were tested by means of multiple-choice tests and as text questions. With regard to reliability, the overall score formed from all exercises (first evaluation perspective) was sufficiently high (Cronbach’s Alpha = .69; theoretical maximum: .72). In the second evaluation perspective, the reliability values ranged between .46 for Question (3) and .86 for Question (5) (Fig. 9). The exam was held about two weeks after the last lecture and was identical in time, duration and content for both courses.

Results and discussions

Qualitative findings

The execution of the CPR in the experimental group went better than the lecturer had expected. This was certainly also due to the high discipline of the students in this dual study, in which attendance is compulsory. The review of the manuscripts was of good quality and the specifications from the Guide of Authors were apparently implemented to a high degree. As far as the review reports are concerned, about 75% of all students had completely answered the questions according to the rubric. Perhaps the judgements could have been given more detailed reasons. In only three cases did the lecturer himself subsequently need to include obvious mistakes from the manuscript in the review. In this context, Kelly (2015) notes that only half of the participants found the review result they received helpful and therefore more reviewers were needed. However, in the revision phase of the study on hand, most of critiques were given due consideration while revising the manuscripts. Even though the script and review itself could not be included in the final grade, the readiness to contribute in the success of the Class Journal was comparatively high. This finding is in contrast to van den Berg et al. (2006) who stated that revisions were insufficiently carried out.

Effect of CPR on personal attitude

From the feedback in the post-test it was possible to derive insights regarding the impact of CPR on personal attitudes. This was measured by the previously introduced 6 variables describing the students´ attitudes towards selected topics linked to the lecture. Each variable was divided into 4 to 5 sub-questions which responses were given on an 11-Likert scale. These sub-questions are exposed by the Fig. 2 and by the Figs. 3, 4, 5, 6, 7, 8, left side. The priority order of the attitudes was visually depicted in the graph on the right. The questions were arranged in increasing order of item means. Different reactions between the groups were statistically evaluated by ANOVA t-test on the means and by Cohens-d. The interpretation was in the light of the five research questions from Sect. 1. In all analyses performed, the Levene-tests were not significant, resulting in variance homogeneity.

Fig. 2

Overview on variables in the Post-Test for evaluation of the CPR-effect

Fig. 3

Post-Test results for Variable (1). Significant (p ≤ .05) for question 1.2, horizontal line shows theoretical maximum, dotted lines indicate scattering of values

Fig. 4

Post-Test results for Variable (2). Not significant (p > .05) over all questions, horizontal line shows theoretical maximum, dotted lines indicate scattering of values

Fig. 5

Post-Test results for Variable (3). Not significant (p > .05) over all questions, horizontal line shows theoretical maximum, dotted lines indicate scattering of values

Fig. 6

Post-Test results for Variable (4). Significant (p ≤ .05) for questions 4.2 and 4.3, horizontal line shows theoretical maximum, dotted lines indicate scattering of values

Fig. 7

Post-Test results for Variable (5). Not significant (p > .05) over all questions, horizontal line shows theoretical maximum, dotted lines indicate scattering of values

Fig. 8

Post-Test results for Variable (6). Not significant (p > .05) over all questions, horizontal line shows theoretical maximum, dotted lines indicate scattering of values

Variable (1): acceptance of learning materials

Acceptance of the instructional environment refers to tools such as scripts, presentation slides, and the software. There were 5 sub-questions on this. The evaluation clearly revealed that the software was practiced significantly more often by the experimental group (Fig. 3, Question 1.2; t(55) = − 2.87; p = .005 (s.); Cohens-d = .80). The cross-check between men and women in the experimental group showed that both used the software equally often (p = .64 (n.s.)). Figure 3 indicates that the ordinate value of question 1.2 for the experimental group is comparatively higher. This means that the students here feel that they used the software more often. Hence, CPR proved to be an effective instrument for the targeted linking of the classroom lecture with the self-learning phase. It was also examined whether CPR increased the use of the standard learning resources as scripts and presentation slides. In the comparison between the two groups regarding the frequency of use of the presentation slides, the experimental group only narrowly failed to reach the significance limit (Fig. 3; Question 1.4; t(58) = .446; p = .07 (n.s.); Cohens-d = .50). The script was the second most often used by both groups (Question 1.3). A within-group-analysis revealed that there was no statistical difference between men and women in the CPR-taught group. However, it was significant that men had more often consulted further literature during their studies than did women (t(23) = 2.66; p = .014 (s.); Cohens-d = .80, not shown in Fig. 3). Overall, the research question (1) in the introductory section is answered positively and the CPR indeed contributed to a significantly higher engagement of students in the software application and in using additional literature at least on the part of men from the experimental group. Further, this section again highlighted the similarity of the two groups with regard to criteria which are not directly related to CPR, such as the use of standard teaching materials as slides and scripts. This shows all the more that CPR has an additional effect on the individual learning situation. The finding extend the research of Stark et al. (2010) because CPR as a PBL-instrument motivates students to take further autodidactic measures even if not directly linked to the CPR-connected instruments such as additional literature from the campus library or from the internet.

Variables (2): experiencing autonomy and (3): subjective success in learning

Figure 4 indicates that there were no significantly different reactions between the two groups. Experiencing autonomy was perceived exactly the same by both cohorts across all Questions 2.1 to 2.4 ($\overline{\mathrm{X}}$ _Experimental = 5.065; $\overline{\mathrm{X}}$ _Control = 5.07). This can be derived from the fact that both lines in Fig. 4 run close to each other. A possible explanation for this could be that the groups assessed their autonomy too much in the light of the classroom lecture, which was carried out identically in both groups. As far as the next criterion on the subjective learning success is concerned, the means over all Questions 3.1 to 3.4 were $\overline{\mathrm{X}}$ _Experimental = 4.944; $\overline{\mathrm{X}}$ _Control = 5.615 which as well were not significantly different from each other (Fig. 5). Here it can be assumed that the impact of CPR on students is not strong enough to evoke a differentiated autonomy experience between both groups.

Also, the results on Questions 3.1, 3.3 and 3.4 reveal that both cohorts estimated the contribution of the lecture Building Physics to the overall study similarly high, but slightly higher in the comparison group. However, the difference is not significant neither across all these questions nor for each of them. An explanation for this could be that the control group already showed in the pre-test a comparatively higher level of background knowledge on this course subject.

Variable (4): group dynamics

The relevant literature on CPR generally confirms a stimulating effect on the feeling of togetherness. In the current study the Variable (4) Group Dynamics was also represented with 4 sub-questions in the questionnaire design and there was a significant difference between the groups. In Fig. 6, Question 4.2, the ordinate value for the CPR-group lies far above the respective value of the control group and therefore CPR presumably gave the experimental cohort a much stronger feeling that the class as a whole would perform better in the exam (Fig. 6, Question 4.2; t(60) = − 3.98; p = .001 (s.); Cohens-d = 1.0). When drawing a gender-specific comparison between the responses for Question 4.2 among both groups it was found that the stimulus coming from CPR was significantly more pronounced for men than for women (t(38) = − 6.48; p = .001 (s.); Cohens-d = 2.0; not shown in Fig. 6). Further, the impression that the group had achieved something together, as investigated by Question 4.3, was again more distinct in the experimental group (Question 4.3; t(60) = − 4.96; p = .001 (s.); Cohens-d = 1.28). Here the effect was significantly stronger for both men and women as compared to the control group (men: t(38) = − 2.23; p = .001 (s.); Cohens-d = .71; women: t(17) = − 2.14; p = .047 (s.); Cohens-d = 1.0); not shown in Fig. 5). The wish to work together in groups again in the future (Question 4.4) was equally high but not significantly different between the courses. Similarly to Question 4.2, a gender-specific analysis between men and women showed that CPR made the wish for more cooperation among students more pronounced on the part of women (t(28) = − 2.15; p = .041 (s.); Cohens-d = .87; not shown in Fig. 6).

Overall, Variable (4) Group Dynamics brought to light that the CPR-application made the test group more self-confident but also that the effect is gender specific. Women in particular would like to repeat this experience of mutually supporting one another whereas men assessed the effects from CPR more target-oriented in terms of better course performance. The research question (3) from the introductory section as well is positively answered. The findings also extend the work of Stark et al. (2010) who already reported a general increase under PBL in motivation and more social involvement which, however, was a cross-gender finding.

Variable (5): experiencing competence

Again, there were no significantly different reactions between the two groups for Variable (5) over all 4 sub-questions. Nevertheless, the result from Question 5.2 (Fig. 7) for the comparison group correlates well with the results from Fig. 4, Question 3.1, as regards subjective learning success. Obviously, for the comparison group the course Building Physics again was more meaningful for their further studies. This is in line with findings from the placement test (Table 1, Criterion 1.5) where there was a weak significance between the cohorts in thematic experience in favour of the control group. Stimulating effects from other lectures might have played a role in this also, but these can hardly be quantified afterwards. Nonetheless, this raises the question of whether CPR was at all too weak to allow the experimental group a thematic superiority over the control group, which obviously had a slight lead in experience regarding Building Physics. Section 3.4 will put more light on this question when the final grades are assessed.

Variable (6): acceptance of the teaching-method

This variable examined whether CPR could be effectively integrated into the current teaching environment of Dual Universities from the students’ point of view. There were 4 sub-questions to be answered. As can be seen from Fig. 8, both lines are close to each other not translating into specific differences although, on the part of the experimental group, the results were slightly superior for Questions 6.2 and 6.4. However, there was again a gender-specific significant difference between the groups. Women wanted comparatively more often that the achievement from the self-study period, i.e. the individual Paper quality, should be taken into account in the final exam (Fig. 8; Question 6.4; $\overline{\mathrm{X}}$ _Men = 3.95; $\overline{\mathrm{X}}$ _Women = 7.75; t(28) = − 2.64; p = .013 (s.), Cohens-d = 1.13; Note: only cross-gender results are depicted here). Hence, CPR could create incentives in particular for female participants, who have so far been rather underrepresented in such engineering courses. Obviously, the risk of failing the examination in such lectures has so far been viewed as being high. CPR may reduce this to a lesser extent by shifting part of the examination towards the self-study phase. This reinforces and expands the hypothesis of Lyman and Keyes (2019) that CPR helps to reduce students’ anxiety and they become more active when participating in the class. The remaining three sub-questions proved to be of no statistical relevance. Together with the predominantly positive experience with CPR implementation, research question (4) as well is answered positively and it can be postulated again that the CPR method positively increases the situational perception of women with a more sustainable effect.

Pre- and post-test comparisons

The question of significant attitude changes arising from the CPR only concerns the experimental group. Results from the pre-test, Variable (2.3) “I prefer to learn alone” (Table 1) were crossed with the 4 sub-questions on Variable (4) Group Dynamics in the post-test (Fig. 6).

Table 2 shows an average improvement of the individual attitude towards the class group belonging in the pre-post comparison in all cases. However, the difference was only significant for the criterion (4.3) “We did something together” (t(56) = − 4.41; p = .001 (s.)). Here, as well, it can be seen that the experimental group developed through the CPR towards an increased sense of togetherness as already described by Stark et al. (2010) and revealed by Sect. 3.2.3. As already found out, this effect results more from the positive experiences of the women. Hence, also this section underlines the implications of this paper, according to which CPR, as practiced here, should mainly be used as a group dynamic instrument.

Table 2 Significance between Pre-Test Variable (2.3) and Post-Test Variable (4.1 to 4.4)

Effect of CPR on the lecture exam

Variable (7) concerned the final written exam which consisted of eight main questions with a total of 21 sub-questions. Of these, 14 were knowledge-based and rather theoretical and 7 were of a practical relevance. With this 2/3 to 1/3 split, the specific focus on repeating and reproducing knowledge in the 3rd semester of the Bachelor’s programme was to be taken into account. However, with a view to the future semesters, application and analyses should also be trained. Therefore, the practical exercises mainly used screenshots from the course software on the basis of which the students should make statements about scenario-based change processes. In order to achieve maximum objectivity in exam evaluation, 30% of the questions were multiple choice, 30% required written answers and 40% calculations. The correction of the exam was done by the lecturer without knowing the group belonging and the individual names but only on the basis of the matriculation number. This prevented any personal influence on the correction.

Figure 9 shows the results of the two groups over the 21 items ranked in increasing order by the degree of achievement of the highest possible score. As can be seen, the respective levels of difficulty were perceived as similar by both groups, which is reflected in an almost parallel progress of the performance curves. The correlation can therefore be assumed to be high. The significance test for the mean values showed no difference in terms of conceptual knowledge between the groups (Questions 1–14; $\overline{\mathrm{X}}$ _Control = 73.71; $\overline{\mathrm{X}}$ _Experimental = 73.07) and both cohorts remained unspecific (t(26) = .08; p = .939 (n.s.)). As depicted by Fig. 9, the CPR-based group tends to outperform the control group in the practical questions, however this effect was not strong enough (Questions 15–21; $\overline{\mathrm{X}}$ _Experimental = 69.57; $\overline{\ \mathrm{X}}$ _Control = 62.86; t(12) = −.69; p = .505 (n.s.)).

Fig. 9

Effect of CPR on exam results as Variable 7. Not significant (p > .05) over all questions, horizontal line shows theoretical maximum

The statistical evaluation of the exam results shows a slight superiority of the experimental group in application-oriented knowledge. Since the tasks in this area had a considerable relation to the training software, it seems plausible that CPR should have contributed to the development of corresponding know-how. This was already confirmed by the evaluation in Fig. 3, Question 1.2, according to which the software was used significantly more often by the experimental group in the self-learning phase. Nevertheless, no clear increase in benefit can be proven in the examination results. This finding supports the statements of Crowe et al. (2015) that the grades were not associated with the Class Peer Review. There is reason to suspect that the effect of the CPR with only one reviewer on the outcome is too weak. Similarly, Aziz and Hossain (2010) found that CPR had only a little effect on the final grade. Rouhi and Azizian (2013) tested two groups against each other, one only receiving corrective feedback and the other only giving the feedback. They found that the former group built up more knowledge than the latter was able to demonstrate in the final examination. Even though in the current study the critiques were mostly addressed in the revision phase, there is reason to suspect that this was carried out without major cross-checking. This even more justifies the approach of Gordijn et al. (2018), who allowed every author to appeal against a critique of the reviewer. Unfortunately, no two cohorts were compared in this study, so that the effect is hardly to be quantified afterwards.

Further general findings

The implementation of the CPR lasted 10 weeks and required considerable effort in advance. This included the installation of a website, preparation of the CPR presentation for the first meeting, the calculation of the deadlines for manuscript, review and revision, the list of reviewers, the checklist for the review, the design of the cover page for the journal including foreword, the consolidation of the revised manuscripts into the finished journal and the presentation of the results in the final course. The regular mail exchange for a single review comprised 3 contacts per student and some reminder e-mails for late participants. Also for some manuscripts smaller improvements had to be made, which were mostly due to compatibility problems of the text-software of the lecturer and the respective student. Each manuscript had to be made anonymous, i.e. the name had to be deleted and the document stored correctly in the respective folder. With 31 participants, these activities were sometimes lengthy and complex. It should be noted, however, that if the CPR is repeated in subsequent academic years, the materials can be reused almost unchanged.

These experiences coincide with statements from the relevant literature on CPR. For example, Tighe-Mooney et al. (2016) refer to a high effort to prepare the students for the procedure and to ensure a smooth process. If more than one review is carried out in order to increase the effect size of CPR, as proposed by Cho and Schunn (2007), then a web-application becomes more meaningful after all. Further, Crowe et al. (2015) doubt that a CPR as an in-class variant could at all enrich a course because it takes up too much time of the classroom lecture. Therefore, the present study applied CPR out-of-class. In fact, it was found that this does not significantly reduce the amount of time taught except for the introductory presentation in the first lecture. Thus, the out-of-class peer review appears to be an effective measure to extend the teaching into the self-study phase.

According to the author, the high effort is justified, even though the final scores did not improve significantly. On the one hand, the CPR has markedly increased group dynamics and substantially improved the collegial atmosphere. This is a positive development for the next semesters and may pave the way for more cooperation. On the other hand, the course was also further educated in academic scientific writing beyond the field of Building Physics which is an effect already proved by Leijen (2017). In addition, the students also acquired experience in giving constructive feedback but also to accept and learn from critique (Şahin 2008). Ross (2006) furthermore sees a valuable insight gain for the lecturers themselves from reading the manuscripts and reviews. This provides valuable impressions on the knowledge level of the course. The author of the present study goes one step further because the manuscripts were largely influenced by previous internships of the students in their companies. In some cases, the students really amazed with their ideas for the application of the training software, so that the lecturer was able to take along some interesting impulses for future teaching. Zejnilagić-Hajrić et al. (2015) finally point out the considerable optimization potential for CPR when used frequently and more intensively. Initial implementation problems should be resolved quickly and benefits increased. In the author’s opinion, even a single Class Paper, i.e. a CPR without a review, is an ideal instrument for requiring students to actively engage with learning content in addition to the classroom lecture. The focus here would then be on consolidating what has been learned and on advanced training in academic writing (Tighe-Mooney et al. 2016). Nevertheless, the study by Crowe et al. (2015) points out that students with a review performed better than those who wrote a manuscript but were not reviewed. According to the author, the CPR thus has overall potential to demonstrate a positive benefit balance for teaching at Universities.

Further, this study demonstrated that CPR also works well outside the class in combination with a fictive class journal. The more the CPR resembles a realistic role play with authors and publishing house, the stronger a motivational effect is supposed to be. This is also where a web application can reveal its potential if the following points are implemented: (1) This tool should automatically address students with standard texts which in this study has already brought noticeable reductions in the editor’s workload. After the study had been carried out the website stats proved that students regularly visited the password-protected download area for gathering information. In addition, also possibilities for uploading manuscripts and their automatic forwarding in an anonymized version should be implemented in such tools because, as previously stated, the execution of the CPR required time-consuming individual forwarding of the manuscripts and review reports. (2) This tool should make it possible to repeat the review several times according to the needs of the lecturer. This point was also raised by Cho and Schunn (2007) who found an optimal number of reviewers to be six. (3) The system should be expandable by the possibility of contradicting unjustified critiques and where a subsequent statement by the reviewer can be given (Xiong et al. 2010). (4) Finally, it should also overcome compatibility problems between the authors’ and the lecturer’s text processing software and enable smooth consolidation to the final journal. This requirement enlarges the current state of research on CPR to the extent that the formal design of the written manuscripts for the subsequent review has so far hardly represented an evaluation criterion. Feedback was often given verbally and only on the content of the essays (van den Berg et al., 2006; Tighe-Mooney et al. 2016). In this study, however, this was different because the students had to consider strict design criteria in their manuscripts. Since the review took place in the absence of the author on the basis of the essay sent by e-mail, incompatible text programs can give the reviewer the impression that the authors disregarded these requirements. (5) After all, particularly for engineering courses, the e-tool should enable the partial crediting of the Paper result to the final exam grade. In keeping with Crowe et al. (2015), the effect from CPR on the final grades is weak in technical courses. In order to benefit not only from the motivational advantages of the CPR but also from the commitment of the students in its realisation, the engagement should at least partly be credited to the course grade. The study showed that especially women demanded this. In this context, Sims (1989) already acknowledged the potential of the CPR as a grading instrument. How this grade is calculated according to particular attributes should be the subject of further research. All in all, this section states that the overall effect from CPR is composed of several individual impacts. Their intensity must be high to produce significantly better exam grades.

Conclusions

The present study examined the effects of a Class Peer Review (CPR) process, which was used as a supportive instrument in the area of problem-based learning at Dual Universities with focus on practical relevance. In contrast to previous in-class applications, an out-of-class design was followed. As a general result, CPR obviously stimulates the use of certain teaching materials in a more targeted way. In addition, this method obviously improves group dynamics of the students which gives credit to the findings of Tighe-Mooney et al. (2016) who identified a stronger group bond. As with Hossain and Tarmizi (2013), the effect of CPR on test results was not gender-specific. However, the present study goes one step further and examined whether gender at least affected the personal attitude of the test subjects. The gender-differentiated evaluation of the results revealed new general insights concerning CPR because more women than men believed that this method is a welcome measure in future learning. Obviously, this can inspire female students in particular to study technical disciplines. Moreover, if a partial grade from the paper is credited towards the written exam grade, this all the more seems to increase interest in engineering subjects. In addition, his research project showed that CPR does indeed offer potential for the integration of e-tools, which supports a thesis by Leijen (2017). These would simplify the complex execution process or allows an even more diverse design with reference to a specific role play.

In keeping with Crowe et al. (2015), the exam grades and CPR were not associated. However, this shows that the overall effect of CPR is composed of several individual impacts. Most probably the embedding of this method in a role play with a class journal as an incentive is one of the factors that raised the average scores in the CPR group at least on the practice-related questions.

In contrast to previous research in this area, this study carried out the CPR out-of-class. Based on the 5 questions in the introduction, the current state of research on CPR can be expanded as follows. First of all, no significantly better final grades can be derived from the performance outside the classroom, i.e. in the private self-study period of the students. Thus, it obviously does not matter whether CPR is carried out within or outside the higher education-specific learning environment. Conversely, this shows that in order to relieve the scarce class hours at Dual Universities, a CRP should rather be carried out during the self-study period without this having a negative effect on the final grade. This study for the first time was also able to derive a gender-specific impact from CPR extending previous studies that only surveyed gender for the equivalence test of comparison groups. Finally, this study also focused on the successful application of exercise software from the engineering field. A CPR in the form of a written feedback has so far only been tested in scientific disciplines such as mathematics, chemistry and physics and the effect has been evaluated on the basis of variables on the personal attitude of the students and their final grade. The specific reference to a certain teaching and learning material, such as the software here, is new. As a result, it was found that CPR can support teaching in a targeted way and noticeably relieve the workload of the lecturer.

Finally, the present study also encounters some limitations which not yet allow any claim to generality for some of the findings. As in almost all CPR studies, the effect of the method is a combination of many individual factors. The present study does not clearly show the effect strength of the special learning environment as a role play in connection with the web platform. Likewise, the effect of the class journal cannot be assessed separately. It is also still unclear whether the announcement of the CPR in the course alone already produces better student performance and how much can be attributed to the review itself. In the latter case, the optimal number of reviews still has to be determined. A comparison with other studies without role play is hardly possible, because these either applied the CPR in non-technical subject areas or did not carry out a pre-post-test comparison at all. Another positive aspect of the study is that the students already had work experience and may therefore have shown a comparatively higher willingness to perform. This indicates that research on cause-effect relationships in CPR is still in its infancy, but that the topic is also very complex because teaching takes place in many different areas and institutional environments. Finally, experiments on CPR are very time-consuming, which to date keeps the number of studies low.

Further studies are planned to quantify the difference of CPR between purely engineering and economics disciplines. The effect from a greater number of reviews will be tested and if reviews from students in higher semesters on manuscripts from lower semesters can be effectively carried out in teaching. If positive findings could be derived from this, students from higher semesters may enrich the general teaching with their own study experiences.