Direct and indirect instruction in educational robotics: a comparative study of task performance per cognitive level and student perception

Abstract

Educational Robotics (ER) has emerged as one of the tools to improve STEM learning in primary education if students are properly instructed. However, there is a lack of studies that guide teachers on which type of instruction should be used for ER in STEM between direct (DI) and indirect instruction (II). As a result, the present study aims to compare the two types of instructions in terms of their effect on learning outcomes, students’ perceptions, and students’ gender differences. We adopted a quasi-experiment comparative research design involving 100 ninth-grade students (13–14 years old). We collected data through achievement tests and perception questionnaires and analyzed them using Cochran’s Q-test, Mann–Whitney U-test, and independent samples t-test. Results show that the group in which II was used performed better than those from the group where DI was used. Also, the results show that girls performed better with DI than boys. Furthermore, students perceived ER as useful for developing collaboration and interest in STEM. Therefore, teachers should be supported in learning how to use II and DI strategically in ER to enhance STEM learning.

Introduction

In the last few years, the synonym for modernization of teaching and education is the application of various educational technologies in teaching. Various digital educational technologies are becoming a fundamental part of students’ lives (Çelik & Yangın Ersanli, 2022). Educational robotics (ER) is one of the most widely represented educational technologies that was introduced in many educational contexts as a modern, innovative learning and teaching tool (Anđić et al., 2023; Caratachea et al., 2023; Grujicic et al., 2016; Leoste et al., 2021; Madariaga et al., 2023; McCormick & Hall, 2022). Many studies indicate that the application of ER in education contributes to the development of 21st-century skills, such as creativity, collaboration, decision-making, problem-solving, critical thinking, and digital literacy (Coşkun & Filiz, 2023; Negrini & Giang, 2019; Romero et al., 2017). For instance, the meta-analysis conducted by Benitti (2012) and Karim et al. (2015) shares similar observations that ER can contribute to the academic achievement of primary students in STEM (science, technology, engineering, and mathematics). Despite the benefit of ER in primary education, a lack of operational framework exists to support teachers in using it (Chevalier et al., 2020; Yang et al., 2020). Some studies have shown how direct (DI) and indirect Instruction (II) could benefit learners in primary schools in different ways (Atmatzidou et al., 2018; Chevalier et al., 2020; Clark et al., 2012), which implies the complementarity of the two approaches. However, the evidence comes in isolated classrooms in a way that can make it challenging for teachers to accommodate the two approaches simultaneously in their practice. The purpose of the current study is to examine the contribution of implementing different types of instruction—DI and II in ER to primary school students’ knowledge of robotics (achievements), as well as the difference in the contribution between the application of DI and II in ER to the students’ perceptions of how these types of instructions contribute to their: enjoyment, interest in learning, anxiety, STEM knowledge and collaboration between the students. This empirical quasi-experimental research contributes to the literature with several parts. First, this study provides information on the contribution of DI and II to students’ knowledge of robotics at different cognitive levels. Secondly, this study provides insights into students’ perceptions about applying DI and II in ER. Thirdly, this study describes gender differences in the contribution of DI and II to students’ robotic achievements and perceptions.

The next sections provide insight into the research; then, we describe the methods, then the results, and finally, to conclude, we discuss the results and their implications.

Theoretical framework

Educational robotics in primary schools

Educational robotics (ER) can be defined as an activity that aims to create an environment where participants learn the basics of robotics through creativity and experimentation (Seckel et al., 2023). The same authors point out that an educational robot is any robot built and programmed by its creators in the learning process using any programming language. Even though scientific research on ER has increased in recent years the idea of using robots in education is over 50 years old (Chevalier et al., 2020; Papert, 1980). However, ER’s active classroom use started over twenty years ago (Leoste & Heidmets, 2019; Ospennikova et al., 2015). Since then, ER has been considered one of the educational technologies that contribute most to the achievement of STEM principles in the classroom (Khanlari et al., 2015; Li et al., 2022). Consequently, ER has become one of the interesting topics among researchers in STEAM education.

Research findings suggest that using ER in primary education can contribute cognitively, affectively, and socially-communicatively. Within the cognitive domain, researchers have shown that the application of ER improves student learning outcomes in general (Atman Uslu et al., 2022), as well as student achievements in STEM learning environments (Benitti, 2012; Darmawansah et al., 2023). The most recent meta-analysis on this topic by Darmawansah et al. (2023), summarized the results from 39 studies (from 2012 to 2021) that dealt with the contribution of ER application in education. The results of this meta-analysis showed that 23 studies were concerned with determining the contribution of ER to student learning, mostly in a problem-based learning environment (15 studies). Students had the opportunity to make—design robots (Spolaôr & Benitti, 2017), develop technological skills (McDonald & Howell, 2012), as well as computational thinking strategies (Leonard et al., 2016), which also influenced the improvement of their knowledge in these areas (Darmawansah et al., 2023). In addition to the above, in the studies of Barak and Assal (2018), Jaipal-Jamani and Angeli (2017), and Konijn and Hoorn (2020), the contribution of robotic STEM activities to conceptual knowledge within STEM disciplines, which is necessary for the development of individual competences within STEM, was examined. The results of these studies show that all students can acquire basic knowledge (show better achievements in scientific concepts on the post-test (Jaipal-Jamani & Angeli, 2017; Konijn & Hoorn, 2020) in this area, but that only a few of them can solve complex problem-based projects (Barak & Assal, 2018). In addition to what was mentioned in previous research, it was observed that the usage of ER can improve student achievements in mathematical knowledge (Hussain et al., 2006) and science knowledge (Karahoca et al., 2011). It has also been shown that the application of ER contributes to the development of functional and transferable knowledge from one teaching subject to another Eguchi (2016), which demonstrates the principles of STEM education and significantly improves students’ creativity.

ER, in addition to the cognitive, can also strengthen the affective domain (Darmawansah et al., 2023). In a meta-analysis by Belpaeme et al. (2018), who reviewed 75 studies on this topic and Darmawansah et al. (2023) in which 23 studies dealt with this issue within the application of ER, it was observed that students perceive social robots to the greatest extent positively, because they had positive experiences learning with them. The perceptions of teachers and parents on this topic are a little more cautious and reserved. In a meta-analysis by Zhang et al. (2021), it was observed that the application of ER has a positive effect on students’ attitudes toward STEM activities (the effect of this influence is small, SMD = 0.01). In addition to the above, it has been observed in numerous studies that ER affects the development of positive student attitudes toward the STEM approach, as well as the development of student motivation and interest in learning on a general level (Anđić et al., 2015; Karahoca, 2011; Lathifah et al., 2019). Numerous studies have also indicated that ER can strengthen students’ interest in academic and technological disciplines, which students were not interested in before using ER (Anwar et al., 2019). ER can also strengthen and develop collaboration among students and improve students’ communication skills (Lathifah et al., 2019; Scaradozzi et al., 2015; Yuen et al., 2014).

However, in addition to these positive contributions of the use of ER in primary education on variables from cognitive and affective domains, an extensive meta-analysis by Seckel et al. (2023), Tikva and Tambouris (2021) and Uslu et al. (2022), show that there is a limited number of research i.e. very little is known about the approaches, pathways, and instruction that should be incorporated into teaching when ER is used in primary schools. Based on the reviewed literature, it is clear that ER can enhance holistic learning among students in STEM in primary education. However, these studies are silent on how teachers can implement ER successfully. Consequently, investigation of the suitable teaching instructions for ER in the present study makes this study more valuable in the research world.

Direct and indirect instructions in education robotics

Minimal or low and strong or high level of guidance, as well as direct (DI) and indirect instruction (II), are terms that are very often confused in ER. Therefore, at the beginning of this part of the paper, let us define the similarities and differences between these topics. Guidance is the act of guiding students in acquiring or constructing knowledge, which refers to broader areas of the teacher’s role that involve organizing, managing, providing feedback, and directing the student learning process (Cooper et al., 2010; Vygotsky, 1978). There are different types of guidance in teaching: minimal guidance, high guidance, and absence of guidance. Minimal guidance is characterized by a high level of freedom in student work, finding solutions, constructing, and presenting knowledge (Kirschner et al., 2006). On the other hand, high or strong guidance means a high level of understanding and managing of student activities by the teacher (Jang et al., 2010; Marzano, 1992). The absence of guidance indicates a scenario in which students navigate the educational material independently with little or no help from the teacher. Guidance refers to the broader aspect of organizing learning, while instructional support refers to specific steps and instructions that teachers prepare to achieve learning outcomes with students (Stronge, 2018). Based on the nature of the guidance provided to learners, scholars have classified teaching instructions into two basic types of teaching instructions: DI and II. DI implies a pedagogical approach based on clear, structured, and systematic delivery of information to students. DI characterizes a high or strong level of guidance, which is reflected in the teacher’s presentation—demonstration of the learning content, step by step while providing all the necessary explanations and answers (Maričić et al., 2023). According to Magliaro et al. (2005), DI involves the following six steps: repeating the content of key concepts; presenting the content in small parts; guided practice after each part; providing corrections and receiving feedback; independent practice to check what has been learned, checking what has been learned. The role of the teacher within DI is reflected in the organization, preparation, presentation of information, demonstration of activities (through models, mock-ups, practical work, and experiments, etc.), asking questions, encouraging discussion among students, preparing tasks, and providing feedback and learning support (Bell et al., 2011; Huitt et al., 2009; Maričić et al., 2022a). On the other hand, students can actively listen, follow, or observe, analyze collected information, answer questions, perform tasks independently, and provide feedback to each other and the teacher (Maričić et al., 2022a). II enables students to explore, experiment, and solve problems independently (Pol et al., 2009). II represents a minimal or low level of guidance, which is organized around key concepts within the content, which are presented to students step by step in the form of tasks or activities that they should solve or complete independently (Maričić et al., 2022b). Within this process, students are not completely independent but are offered indirect guidelines within each of these tasks (Lazonder & Egberink, 2013). The teacher’s role is a facilitator; he/she/they organizes, prepares, and creates activities and tasks for students with built-in guidelines and appropriate questions, which provide adequate support to students and guide them in their independent work (Bell et al., 2011). Students, on the other hand, independently research, experiment, devise and construct the way they gain knowledge, collect information, obtain results, ask questions, and make mistakes, which they correct through discussion with their peers and the teacher (Bell et al., 2011; Maričić et al., 2022a). According to Atkin (2016), the main features of indirect teaching are: students are engaged through tasks that they solve independently by exploring different sources of information; in this part, students ask questions, try different activities, experiment, and correct their mistakes. In this paper, we focus on specific steps and instructions that teachers prepare to achieve learning outcomes with students. Therefore, we use the terms DI (high or strong level of instruction) and II (low or minimal level of instruction).

In the ER learning environment, students most often receive II or DI, categorised according to the level of support students receive from teachers in solving robotic tasks (Atmatzidou et al., 2018). This level of student guidance is largely dependent on the outcomes of ER. In a typical ER learning activity, students work on robotics activities with prepared worksheets from teachers who may have varying degrees of guidance (Atmatzidou et al., 2018). As for ER, there is conflicting data on the contribution of II and DI, or their complete absence, to student learning outcomes. When it comes to programming robots in ER, programming with the use of experiments involving II is not sufficient to ensure quality learning. Therefore, it is necessary to use DI in ER teaching (Kirschner et al., 2006; Mayer, 2004). Biesta and Burbules (2003), Chevalier et al. (2020), and Mayer (2004) point out that when II is used, students may use a blind search, which leads to a blind trial and error strategy. This means that students collect random solutions, try them out, and search for the most appropriate one, and in this process may not develop systematic cognitive activity, resulting in a not-so-good learning outcome. Clark et al. (2012), Kirschner et al. (2006), and Sweller et al. (2007) point out that at ER, DI contributes more to the achievement of learning outcomes than II, especially when students have no experience in robotics. However, the same authors point out that II is suitable for practicing students’ acquired robotics knowledge. Chevalier and Giang (2020) conducted a quasi-experimental study involving a control and an experimental group of students. In the control group, students completed the task without imposed constraints and DI, with the option of using different sources of information. The experimental group was subjected to a didactic intervention that included DI and time limits on student activities. These included operating, teacher presentation, programming, and testing the robot in the playground. The results of their research indicate that the students in the control group who had unrestricted access to the robotics activities (i.e., unrestricted access to the programming interface) developed a working and learning approach involving trial-and-error behavior. In contrast, the students in the experimental group, who were given limited time and DI in the use of the programming interface, developed better cognitive processes related to understanding problems, generating ideas, and formulating solutions, and may represent an effective avenue in the ER of primary school students. In a recent literature review, Tikva and Tambouris (2021) cited only 12 studies that investigated what kind of instruction and support primary school students should receive in ER. Chevalier et al. (2020) and Tikva and Tambouris (2021) point out that there is not enough research looking at the type of instruction that should be used in ER of primary school students. That research is particularly limited in the affective domain. In one of the few studies on this topic by the author Edwards (2016), students’ perceptions of the usage of telepresence robots (where the teacher was an instructor—explicit guidance) and social robots (where the robot is in the role of a teacher—implicit approach) were examined. The results revealed that students had significantly more positive perceptions towards the application of telepresence robots with explicit teacher guidance.

Based on their research, we identified the following types of gaps in ER primary students’ knowledge when it comes to applying certain types of instruction: knowledge gaps—lack of studies; empirical gaps—lack of data that could be used to refute or support a particular hypothesis; practical gaps—lack of research that gives teachers practical suggestions for applying it in practice. These gaps were the inspiration for the design and implementation of this study, with the main aim to investigate the contribution of DI and II to primary students’ ER outcomes—achievements and perceptions, according to the research questions posed.

Research questions

This study aims to answer the following research questions:

• RQ1 What contribution does the use of DI and II have on the ER learning outcomes of the primary school students who took part in this study?

• RQ2 To what extent do students’ learning outcomes correlate with their differentiated perceptions of different instructional approaches to learning robotics content, given the evolving landscape of II and DI and gender differences?

In line with the research questions, the methodology of this research was developed.

Methodology

Research design

This research was quasi-experimental and was conducted in a real-everyday-educational environment. Following the recommendations for quasi-experimental research (Cohen et al., 2002; Gopalan et al., 2020), this research was used to investigate the relationship between the intervention and the impact on learning outcomes. The participants in the group were not randomly assigned. The research design followed the recommendations of Cohen et al. (2002) for a quasi-experimental design with group post-tests only. This research did not include a pre-test to test the student’s prior knowledge. Since the students who participated in our study had no prior training or attended any robotics workshops, it was impossible to test their prior knowledge in this area. The following parts of the paper explain the students who participated in the study, the research design, the method of data collection and processing, and the demographics of the students who participated in the study.

Study participants and group dynamics

100 ninth-grade students (13–14 years old), who were divided into two groups, participated in the study. The groups were formed according to the following criteria: number of students (each group consisted of 50 students) and average grade of the student from the previous grade. The overall grade point average (GPA) at the end of the previous grade was determined by analyzing school records, i.e., pedagogical documentation. The average grade point average at the end of the first semester was 3.91 in GDI and 3.72 in GII. Since both groups had many students (50 each), the students were divided into four classes during the lessons. Each class consisted of 25 students working in groups of 5 on one robot set. In this way, the recommendations of previous studies stating that groups of students working on one robot set should include 4 out of 6 students were considered (Barker & Ansorge, 2007; Chiazzese et al., 2019). The division of 100 ninth-grade students into four classes was based on two main factors: the number of students and their grade point average from the previous school semester. Each class had 25 students between whom there was no statistically significant difference in terms of GDI (p > 0.05) from the previous semester. In this way, consistency was achieved between all classes participating in the study.

Instructional design of ER program and implementation process

This study was designed as a quasi-experimental comparative study between students who used DI (GDI: group direct instruction) to learn ER learning content and those who used II (GII: group indirect instruction) to learn the same content. To investigate the effects of DI and II on student learning outcomes, this study was conducted through the following phases:

• Phase 1: Design of an ER program for primary school students—in Montenegro, the country where the research was conducted, there is no compulsory subject of robotics or anything similar. The content of robotics is taught through extracurricular activities. These extracurricular activities are designed by teachers in collaboration with the school administration and offered to students who participate voluntarily. In this study, the teaching content for the extracurricular robotics activities was developed by two computer science teachers who led the program in collaboration with the researchers. In designing the teaching content to be included in the program, the computer science curriculum for primary schools in Montenegro was consulted, then published available research on this topic such as Chalmers (2018), Chen et al. (2017), Gaudiello and Zibetti (2016) and Ileva (2010). Based on the recommendations in the above literature sources, the extracurricular program ER was selected, whose teaching topics and learning outcomes that students should achieve during the learning are presented in Table 1.

  The students who applied for the extracurricular ER activities did not participate in similar classroom or extracurricular activities. The ER extracurricular classes were conducted twice a week for two school hours of 90 min each.

• Phase 2: Call to students—in this phase, primary school students in grade nine (13–14 years old) were invited to participate in the programming ER, which was organized as an extracurricular activity. The students were described the programme ER and decided to participate voluntarily.

• Phase 3: Group formation—The registered students are divided into two groups, where we tried to divide the groups evenly according to (a) the number of students, (b) the success of students in the previous grade, and (c) gender.

• Phase 4: The introductory lessons—in the ER extracurricular classes were the same for both groups of students. During these lessons, the teacher explained the contents of the Lego sets and the purpose of the different parts and demonstrated the programming. The aim was to familiarize the students with the Lego Robotics set and the basics of its use. The introductory lessons in the ER extracurricular classes were the same for both groups of students.

• Phase 5: Introduction of the experimental factor—both groups of students learned the same lessons on robotics. The lessons in all groups consisted of three parts: introduction, main part, and conclusion. In the introductory part of the lesson, the teacher used a creative presentation, lecture, animation, video, or similar to introduce the students to what they will learn in that lesson and did a short brainstorming session with the students on the topic discussed. This activity usually took about 10 min. Then the students implemented the activities, which were in the form of instruction sheets on their tables. In the GDI group, the instruction material was as per DI, while in the GII group, the instructions were as per II. The students then solved the tasks in groups of 4–5 students. This part of the lesson lasted about 65 min. In the last part of the lesson, students presented their work and received feedback from the teacher and other students in a discussion. The last part of the lesson lasted about 15 min.

• Phase 6: Repetition of the newly acquired ER knowledge—1 week after the implementation of the intervention, the teacher renewed the newly acquired knowledge about robotics with the students. This involved students in groups having the opportunity to build a simple robot with parts, motors, and sensors of their choice.

• Phase 7: Testing the students’ robotics knowledge and perceptions—3 days after the review of the acquired knowledge, the students’ robotics knowledge was tested. To test the knowledge in both groups, the same tasks were used—tests that the students solved individually. One week after the test, the students filled out a questionnaire expressing their perceptions about the way they had learned the robotics content.

Table 1 Teaching topics and learning outcomes were realized with students in this study

Development and description of ER instructional materials

The students in the GDI received the instructions directly from the teacher and then repeated them independently in groups. Using a projector, the teacher demonstrated robot constricting or programming the steps step by step, explained the aim of these steps and the students then repeated the process in groups using the appropriate parts of the robot. To apply DI, the teacher used the official teaching materials for the Lego Robot website, which he translated and adapted to the student’s native language. The instruction sheet that the teachers used in this process is shown in Fig. 1.

Fig. 1

DI for students on the teacher’s instructional slide

Unlike them, the GII students carried out these activities in student groups independently from the teacher. They followed the II on the instruction sheets as they worked. The instruction materials provided enough information for the students to complete the tasks independently. An example of these instruction sheets can be found in Fig. 2. In Appendix 1, an illustration of a lesson plan designed for a group with indirect instructions is provided.

Fig. 2

II for students on the student instructional sheets

The type of instruction was the only difference in the education of these two groups. Both groups learned the same teaching content, had the same ER teaching materials, and were taught by the same teacher.

Data collection tools

We collected data related to RQ1 through the achievement test and data related to RQ2 through the perception questionnaire. The following sub-sections include the development and adaptation processes, and validity and reliability studies of these data collection tools.

Test assessment of students’ robotics achievements

To answer the first research question, tests were developed to assess the achievement of the learning outcomes of the students who participated in this study. The test examined the same knowledge about robotics in both student groups. The test items were based on the Bloom-Anderson-Krathwohl taxonomy (Anderson et al. 2001). In creating the tests, the available scientific literature on the subject was consulted. For example, research by Tsai et al. (2021) shows that Bloom’s taxonomy of educational goals is suitable for the development of the self-efficacy scale for primary school students in robotics. Gummineni (2020) and López, et al. (2019) indicate that the Bloom-Anderson-Krathwohl taxonomy is an appropriate approach for developing activities and testing students’ robotics knowledge, leading to more achievement of student learning outcomes at higher cognitive levels. Very important and interesting is the research of Shyr et al. (2019), who, based on experimental research involving experts through Delphi group members, identified the following six levels of knowledge as very important for robotics education: remember, understand, apply, analyze, evaluate, and create. In our study, each cognitive level was examined with one task in each test. The tasks belonged to the following cognitive levels: knowledge, understanding, application, analysis, evaluation, and synthesis-creation.

To test the student’s knowledge, the same parts of the robot used in class to elaborate the teaching content were used. In the tasks on the cognitive level knowledge, the students were asked to name the parts of the robot that were on their desks. The tasks at the cognitive level of understanding put the students in a position to describe in their own words the role of a particular part of the robot sitting on their desk. At the understanding level, there was a task of this type: describe the part of the robot on your desk and its role. The cognitive application-level task required students to demonstrate a certain level of skill. The cognitive application-level task included the following task: Demonstrate (show an example, do the programming) how the light sensor works. The cognitive analysis level question asked students to compare different robot parts. At this level, students were asked to compare, for example, which sensor is better for orienting the robot on a monochrome surface: a light sensor or an ultrasonic sensor. The tasks at the cognitive evaluation level required students to compare the two actions of the robots shown in the video displayed on the projection screen and to judge why one of the robots performed the task correctly, and the other did not. The synthesis-creation cognitive level task required students to creatively use their acquired knowledge to create a new whole or develop a new idea. An example of the achievement test used in this study (translated from the student’s native language into English) is shown in Appendix 2.

Exploring student perceptions

To answer the second research question, a questionnaire was developed to gather the perceptions of the students who took part in this study. The same questionnaire with the same questions and items was administered to both groups. The questionnaire contained sixteen items. The questions were of the closed-ended type. For the given items, students filled in a five-point Likert scale with the following responses: strongly agree, agree, partly agree, disagree, and have no opinion. The questionnaire covered four concepts: Collaboration among students, Contribution to STEM knowledge, Self-confidence, and Interest in learning. The items included in this questionnaire are listed in Appendix 3 and were created based on previous similar research that addressed students’ perceptions of robotics and other educational technologies. To ensure the reliability and relevance of the questionnaire, we used the results and conclusions of several research studies. For the development of the items on the concept of “collaboration between students’,” we used the research findings of Andjić et al. (2019) and So and Brush (2008). For the items on the concept of “contribution to STEM knowledge”, we drew on the studies of authors Oner et al. (2016), Sinatra et al. (2015), and Tseng et al. (2013). The items for the concept of “Self-confidence” were based on previous studies by Alemi et al. (2015), Istikomah and Wahyuni (2018), and Naneva et al. (2020). Finally, the items in the “enjoyment and interest in learning” domain were based on research by Chang et al. (2020) and Pantziara and Philippou (2015). Considering this previous research, our questionnaire (Appendix 3) was carefully designed to gain deeper insights into students’ perceptions of robotics and how different forms of instruction affect students’ perceptions of robotics.

Validity and reliability of the study

The content validity of the developed teaching material—achievement test, and perception questionnaire was first ensured by the opinions of educational researchers, experienced teachers, and students. All developed materials were first evaluated by five educational researchers with more than ten years of professional experience. According to their evaluations, the materials were improved until the researchers reached a complete consensus and agreement on the quality, content, and concept of the developed materials. Afterward, the materials were reviewed by five STEM teachers with more than ten years of experience in teaching and instructing robotics. Following their evaluations, the materials were further improved. Then, these materials were presented to ten students who were not involved in the research, and they were asked to rate the clarity and understanding of the materials. When complete clarity, understanding, and agreement were achieved by these students, teachers, and researchers, the materials were considered valid for implementation. A similar approach to the validation of the developed materials was recommended in previous similar studies (Anđić et al., 2018; Maričić et al., 2019). The Cronbach’s alpha coefficient of the questionnaire was calculated as 0.71 and for the test 0.74, which indicates good reliability (Cohen et al., 2002).

Data analysis

We used Cochran’s Q-test to analyze the differences between the tasks in each group (RQ1). The Cochran’s Q test is a statistical procedure that can be used to examine whether there are differences between three or more similar groups for a dichotomous dependent variable (McGrum-Gardner, 2008). The first assumption for Cochran’s Q test is that each dependent sample has the same number of observations. The second assumption is that having a dichotomous dependent variable. The data gained from the achievement test was represented only by 0 s and 1 s. In this study, for each task, we coded a 1 for students who successfully completed the task and a 0 for students who did not complete the task, or they did it unsuccessfully. We also used the Mann–Whitney U-test, also known as the Wilcoxon rank-sum test, to determine whether there was a difference between groups on task achievements and a difference between girls and boys on task achievements (RQ1). The Mann–Whitney U-test compares two independent groups, which are not normally distributed as in this study. Since the data regarding students’ perceptions about using DI and II in ER is distributed normally, we used independent samples t-tests to determine whether there was a significant difference between the themes and the groups (RQ2). For the statistical processing and analysis of the collected data, the Statistical Package for Social Sciences (SPSS) version 23 was used.

Results

Differences between task accomplishments per cognitive level in groups GDI and GII

The frequencies of tasks’ accomplishments per cognitive level for each group are given in Table 2, and the mean ranks of the groups regarding the tasks per cognitive level are given in Table 3.

Table 2 The frequencies of tasks per cognitive level accomplishments for groups

Table 3 Mann–Whitney U test analysis results

Accordingly, group GII, in which students receive II, is more successful than group GDI at each cognitive level. GII performed better at all cognitive levels. The results of the analysis carried out to examine whether this success is statistically significant are given in Table 3.

We observed a significant difference between groups related to each cognitive level. The category of understanding has a large effect. Group GDI and Group GII tasks per cognitive level accomplishments are given in Figs. 3 and 4.

Fig. 3

GDI tasks’ accomplishments

Fig. 4

GII tasks’ accomplishments

Differences between tasks per cognitive level for each group were analyzed through Cochran’s Q test. Accordingly, we found a significant difference between cognitive levels of understanding and synthesis-creation for Group GDI (Cochran’s Q = 16.827, p < 0.05, df = 5). It is in favor of cognitive level synthesis-creation. We found a significant difference between cognitive levels of understanding and knowledge, understanding and analysis, and understanding and evaluation, understanding and synthesis-creation for group GII (Cochran’s Q = 32.531, p < 0.05, df = 5). It is in favor of cognitive levels of knowledge, application, analysis, evaluation, and synthesis-creation.

The failure rate in tasks on the cognitive level of understanding is higher for both groups and differs significantly compared to other tasks. This difference is significant only between cognitive level understanding and synthesis-creation for group GDI. However, it is significant between understanding and cognitive levels of knowledge, analysis, evaluation, and synthesis-creation for Group GII. So, task on the cognitive level of understanding is difficult for both groups. However, the task on the cognitive level of understanding was the most difficult task for group GII, which received II, because this group did better in all tasks except task three on cognitive level application. At the cognitive level of understanding, students were asked to define concepts in their own words. However, this was the most challenging task for those who received both DI and II. For this cognitive level, task design, content or implementation is an important detail for both instruction groups.

Differences between girls and boys according to task accomplishments per cognitive level

Girls outperformed boys in all tasks except application level. A Mann–Whitney U test was conducted to see if this was statistically significant. The mean ranks of success in task solving per cognitive level of girls and boys and Mann–Whitney U test results are given in Table 4. The results of the Mann–Whitney U test show there is no significant difference between girls and boys related to each cognitive level.

Table 4 The mean ranks of girls and boys and Mann–Whitney U test results

For GDI and GII, it was also examined whether there was a difference between the groups according to gender. Accordingly, there is a significant difference between girls and boys in cognitive level of knowledge and analysis for GDI. For the cognitive level of knowledge, the mean rank of boys was 20.81 while that of girls was 30.58, and this difference was in favor of girls. At the cognitive level of analysis, while the mean rank of boys is 19.77, the mean score of girls is 31.71, and this difference is in favor of girls. For GDI, female students were more successful in solving the tasks at cognitive levels of knowledge and analysis. However, for GII, there was a significant difference between girls and boys for tasks at the cognitive levels of application, analysis, and evaluation. For successful task solving at the cognitive level of application, the mean rank of boys was 30.04 while that of girls was 19.73, and this difference was in favor of boys. Results indicate that for successful task solving at the cognitive level of analysis, the mean rank for boys was 28.82, while the mean score for girls was 21.27, and this difference was in favor of boys. In terms of successful task solving at the cognitive level of synthesis-creation, the mean rank of boys was 29.21 while the mean score of girls was 20.77, and this difference was in favor of boys. For GII, male students were more successful in task-solving at the cognitive levels of application, analysis, and evaluation.

Analysis of students’ perceptions about using DI and II in educational robotics

Students’ perceptions about training were examined under four themes: collaboration, STEM knowledge, self-confidence, and enjoyment. The descriptive statistics of the themes are given in Table 5.

Table 5 Descriptive statistics of the domains of students’ perceptions

Students think the robotics activity contributed most to their cooperation with other students. This is followed by self-confidence (Fig. 5). An independent samples t-test was performed to determine if there was a significant difference between the mean scores on the themes by gender and group.

Fig. 5

Students’ opinions about ER activities

When the mean scores of girls and boys are analyzed, it is seen that girls have higher mean scores than boys for collaboration, self-confidence, and enjoyment, except for the STEM knowledge theme. As a result of the analysis conducted to examine whether this difference is statistically significant (Table 6), it was found that there is no significant difference between girls and boys in terms of STEM knowledge, self-confidence, and enjoyment themes.

Table 6 Independent samples t-test results regarding gender

Students in the GII group had higher scores for each sub-theme than students in the GDI group. This difference is statistically significant between GDI and GII in terms of STEM knowledge, self-confidence, and enjoyment themes (Table 7). Accordingly, GII has higher self-confidence than GDI during the activity. In addition, in terms of STEM knowledge, GII students think that robotic activity contributes more to their knowledge than GDI. Finally, GII students had more fun in the robotics activity than GDI students.

Table 7 Independent samples t-test results regarding groups

Discussion

This research used a quasi-experimental research design to investigate the contribution of DI and II to students’ achievements and perceptions of robotics. The results of our research show that there is a significant difference in success in solving robotics tasks at different cognitive levels within the two instructional groups—GDI and GII. This suggests a different contribution of DI and II to students’ robotics knowledge. The results of our study indicate that the students in group GII who received II performed better in solving the tasks at all cognitive levels. This difference between the groups (GDI and GII) is also statistically significant except for the level of understanding. This suggests that the students who acquired their robotics knowledge by using II acquired better knowledge than the students who acquired it by using DI to solve the following robotics tasks: Identify parts of the robot; demonstrate how a particular part of the robot, such as a sensor, works; compare two robot sensors and make a good choice which to use; evaluate the robotic construction and robotic performance of tasks and make suggestions for their improvement; and independently construct and program a simple robot. The results of our study contrast with the findings of Biesta and Burbules (2003), Chevalier et al. (2020), and Mayer (2004), who suggest that students may spend a lot of time searching for information if they are not given direct instructions on ER, which leads to student distraction and poor performance. One of the reasons for the better performance of students who used II in learning robotics content could also be the design of the instructions. In our study, II was designed for the students not to solve the task for them, but to provide enough information for them to solve it independently. Thus, students were prevented from wasting time searching for information that would distract them. Yet they actively participated in developing solutions to the tasks and in solving them. This assumption is consistent with research by Prince and Felder (2006) and Rueuetmann and Kipper (2011), which suggests that well-designed II contributes more to the achievement of primary school students’ learning outcomes in technology and information literacy than DI. However, students in the GII group who obtained knowledge using II acquired better knowledge at all cognitive levels. This suggests that it is necessary to combine the features of II and DI when covering robotics content in primary school. This recommendation of ours is supported by previous research by Chevalier et al. (2020), which indicates that combining DI and II in robotics education of primary school students would achieve the best learning outcomes. Based on the results of this study as well as the results of the previous studies mentioned above, a recommendation for practice can be made that when designing II in ER for primary school, a brief description of the robot parts should be included so that students acquire this knowledge. The results of our research show differences in the contribution of DI and II regarding the gender of the students. Girls in the GDI demonstrate a significant advantage in cognitive knowledge and analysis, while boys in the GII approach demonstrate improved cognitive abilities in application, analysis, and synthesis-creation. These results illuminate the complex interaction between instructional methods, gender, and cognitive outcomes in robotics education. Our results indicate that girls achieve better robotics knowledge to the extent that they acquire it with the application of DI, while boys achieve better knowledge to the extent that they acquire the same knowledge with the application of II. Our data are supported by similar previous research such as Lufkin et al. (2014), Sullivan et al. (2016), who indicate that girls achieve better learning outcomes in terms of how much robotics and technical knowledge they acquire with the application of DI and strong guidance. Our results, supported by the cited literature excerpts, provide significant information to teachers when creating instruction for ER.

Investigating students’ perceptions of learning approaches and their impact on collaboration, STEM knowledge, confidence, and enjoyment of learning provides valuable insights into the experiential aspects of learning. In addition, the increased confidence reported by students after participating in robotics activities is consistent with research suggesting that hands-on experiences and problem-solving opportunities can significantly increase students’ self-efficacy. When students overcome challenges and complete tasks, they gain more confidence in their abilities, which contributes to greater engagement and persistence in learning. The findings highlight the importance of collaboration as a key element for engagement in robotics activities. This is in line with previous research by Caratachea et al. (2023), Madariaga et al. (2023), and McCormick and Hall (2022) who suggest that ER contributes to students’ confidence, problem-solving, efficiency, and creativity. This study also reveals gender differences in perceptions of the contribution of ER. Girls reported higher levels of confidence and enjoyment compared to boys, but not to their STEM knowledge. However, boys have a higher perception of STEM knowledge. These findings are like previous research in science—physics (Radulović et al., 2022; Reid & Skryabina, 2002), and robotics (Su et al., 2023), which emphasizes that boys have more positive attitudes towards these sciences and enjoy learning from girls more. For example, Kucuk and Sisman (2020) studied students’ attitudes towards robotics and STEM in Turkey and found that girls were much less enthusiastic and confident about learning robotics than boys. Milto et al. (2002) found that boys were more confident than girls in robotics-related activities. This finding warrants careful consideration and further research. Daniela and Lytras (2019), Konijn et al., (2020), and McDonald and Howell (2012) suggest that family influence, which attracts boys more than girls to technical science, as well as social structure, may influence girls’ more positive attitudes towards technical science. In these studies, organizing events such as Girls in Robotics Days and the like, as well as promoting female role models in technology, can contribute to a more positive attitude of girls towards robotics. Future research should examine these assumptions.

Lastly, students who received II had a higher perception of self-confidence, enjoyment, and STEM knowledge than GDI. Our findings are in line with previous studies such as that of Anđić et al. (2022), Chang and Chen (2020), and de Vink et al. (2022), which emphasize the significant impact of greater student autonomy in exploring robotics and similar technological learning materials. These authors suggest that students who have more opportunities to explore robotic educational content independently, in collaboration with peers, and without explicit instructions from the teacher tend to develop greater self-confidence and enjoyment of learning and develop a more positive attitude towards learning. We hypothesize that the provision of II will allow students a greater degree of freedom to explore robotics curriculum content, contributing to the development of positive attitudes compared to students using DI. Future research efforts should aim to empirically test this hypothesis.

Conclusion

The present study shows that II for ER enhances STEM learning outcomes among primary school students. Besides, DI proved to support some students to perform better in some ER tasks in STEM learning. This shows the need for teachers to use both II and DI at some points in their teaching. A review of the design of the task and the need for a design-oriented study for tasks can shed light on future work. Also, our results indicated there is a difference between girls and boys in terms of being successful in tasks with DI or II. For girls, it is better to receive DI to solve problems and tasks in the ER. This can be explained by the fact that girls are more successful in DI due to their lower self-confidence in computer science activities (Yadav et al., 2014). However, future studies examining this situation are needed. This could be a good solution to educationally regulate gender balance, especially in the field of computer science. Furthermore, students perceived ER as a catalyst for developing collaboration skills and interest in STEM learning. Based on the results, we call for support from teachers to enable the application II and DI strategically in the ER. Also, research should be conducted on how to develop equitable instruction in ER to support both girls and boys successfully.

Limitations

Our research has several limitations. First, our study is quasi-experimental and provides useful insights. However, as suggested by Cohen et al. (2002) and Gopalan et al. (2020), the results of quasi-experimental research should be interpreted with caution due to the lack of randomization, selection bias, and limited control over extraneous variables. Future research should contribute to this knowledge by applying different research approaches. Our second study was conducted with students without prior knowledge of robotics. The effects of DI and II on the outcomes of students with prior knowledge of robotics would contribute differently to learning outcomes. Future research should test this hypothesis.

Appendices

Appendix 1: Simple lesson plan for ER classes with indirect instructions

Appendix 2: Simple example of the achievement test used in this study

Name: _______________	Date: ______________
Introduction: Welcome to the Robotics Achievement Test! This test has been designed to test your knowledge, understanding and practical skills in the fascinating world of robotics. Each section of this test has been carefully crafted to assess different facets of your expertise, from basic knowledge to advanced analytical thinking.
Instructions: Read each question carefully to ensure you fully understand the task. Ensure that your answers are concise and accurate, and demonstrate a thorough grasp of the robotics concepts discussed in class. Feel free to express your thoughts and ideas confidently. This test is not only about correctness but also about showing your unique perspective.
Task 1: The parts of the robot standing on your table are marked with the numbers 1, 2, 3, and 4. Carefully observe and analyze the parts of the robot and then write their names in the appropriate space below: (a) The part of the robot marked with number one is _______________________ (b) The part of the robot marked with number two is _______________________ (c) The part of the robot marked with the number three is _______________________ (d) The part of the robot marked with the number four is _______________________
Task 2: Define and explain the basic functionalities of the robot parts from your table marked with a number 1. In the description elaborate on its significance in the overall functionality of the robot. (Minimum 70 words) ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Task 3: Open the LEGO MINDSTORMS platform on the computer at your workstation. Use this platform to create a simple code for a robot light sensor. Then save the code under your name on the desktop of your computer.
Taks 4: Compare and contrast the suitability of a light sensor versus an ultrasonic sensor for orienting a robot on a monochrome surface. Justify your choice based on their respective characteristics. (Minimum 80 words) ______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Task 5: Watch the provided video displaying two robots performing a task. Compare the actions of the robots and explain why one robot succeeded while the other did not. Support your judgment with relevant observations. (Minimum 100 words) ______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
Task 6: Design and create a simple robot that can orientate itself in space using two or more sensors.

Appendix 3: Perception questionnaire

Concept	Item	Question
Collaboration between the students	Q1	Collaborative learning in my group was effective
	Q2	I felt part of a learning community in my group
	Q3	I actively exchanged my ideas with group members
	Q4	I was able to develop new skills and knowledge from other members in my group
Contribution to STEM knowledge	Q5	I was able to learn about science from the robotic project
	Q6	I was able to learn about Mathematics from the robotic project
	Q7	I was able to learn about computer science from the robotic project
	Q8	I was able to learn about technology and engineering from the robotic project
Self-confidence	Q9	When participating in robotics projects, I felt safe
	Q10	I wasn’t afraid to make a mistake when constructing the robot
	Q11	I wasn’t afraid to make a mistake when programming the robot
	Q12	I easily corrected the mistakes I made
Enjoyment and interest in learning	Q13	I actively seek as much information about STEM in robotics projects
	Q14	I enjoy the uncertainties in my robotics activities related to STEM
	Q15	I am more interested in STEM assignments that include robotics
	Q16	I enjoy doing things related to STEM that are related to robotics