Sustainable development and formative evaluation of mathematics open educational resources created by pre-service teachers: an action research study

Abstract

In the post-pandemic world, UNESCO advocates the development of open educational resources (OER) to remedy the financial pressures of students and their families. We thus conducted a dual-cycle action research study aiming to develop a sustainable model with minimal cost for creating OER for secondary school mathematics teachers and students. Our theoretical foundations drew upon existing sustainable models of OER development and experiential learning theory to enhance the efficacy of OER. Formative evaluation techniques were employed in both action research cycles, where pre-service teachers developed the OER and in-service teachers provided suggestions for improvement. In the second cycle, we tested the use of our OER with 35 Grade 11 students to examine both student achievement and retention. The test results showed a significant learning gain (pre-test: Mdn = 2.00; post-test: Mdn = 10.00), which was retained over 2 weeks (delayed post-test: Mdn = 11.00). However, the students’ performance on more advanced questions was not satisfactory. Based on an overall reflection on the study, we proposed a sustainable model of OER development, which leveraged the manpower of pre-service and in-service teachers and incorporated formative evaluation techniques. Furthermore, we provided recommendations for enhancing the future development of OER in mathematics education.

Introduction

Recently, students in Hong Kong have attained their lowest scores in a territory-wide mathematics assessment in approximately 20 years (Mycroft & Yiu, 2023). This trend echoes findings observed in some other regions globally, where students experienced declines in mathematics achievement after the COVID-19 pandemic (e.g., Battisti & Maggio, 2023; Moliner & Alegre, 2022). As Chen et al. (2022) noted, students from low-income families might have experienced more disadvantages due to financial hardships and inabilities to afford commercial learning materials. Open educational resources (OER) thus play a vital role, particularly in enhancing accessibility, helping students recover from their learning setbacks, and supporting their independent learning. High-quality OER can also serve as supplementary resources for teachers and create more learning opportunities for students (Huang et al., 2020).

As defined by UNESCO (2019), OER “are learning, teaching and research materials in any format and medium that reside in the public domain or are under copyright that have been released under an open license, that permit no-cost access, re-use, re-purpose, adaptation and redistribution by others”. In the post-pandemic world, UNESCO (2020) calls for global collaboration among governments and non-profit organisations to develop and distribute OER and open platforms. As OER provide users free and permanent permission to adapt and reuse the resources, the use of OER ensures equitable education—Sustainable Development Goal 4 set by the United Nations (Hilton, 2020; UNESCO, 2019). In their meta-analysis of OER research, Tlili et al. (2023a) found that the use of these cost-free resources, at worst, did not impair student learning.

However, as OER generate little or no financial benefit, sustainability is a major challenge faced by OER developers (Sousa et al., 2023). There is therefore a need to explore sustainable models that can support the long-term viability and development of OER (Sousa et al., 2023; Tlili et al., 2023b; UNESCO, 2019). Second, not all OER meet the necessary standards of accuracy and pedagogical effectiveness (Tlili et al., 2023a). For example, teachers of Barana et al. (2021) complained that some mathematical terms in their OER were too difficult and not suitable for students. As Perifanou and Economides (2023) observed, most OER in some popular repositories lacked peer evaluation. This quality issue can impair the effectiveness of OER and discourage teachers from adopting them (Tlili et al., 2019). It is therefore essential to ensure the quality of OER produced (Tlili et al., 2023a).

Following the call of UNESCO (2020), our goal is to develop OER that facilitate students’ independent online learning while addressing the challenges associated with OER development. Ideally, students can use the OER either to catch up with learning loss or to learn mathematics knowledge. Hence, this action research study was conducted with two primary objectives:

• Objective 1: To develop a sustainable model for creating OER with the involvement of pre-service teachers (i.e., students enrolled in a teacher preparation programme) and in-service teachers (i.e., qualified teaching staff in schools). This objective explored the potential of leveraging pre-service teachers’ capacity in OER development, with support from in-service teachers.

• Objective 2: To propose strategies for enhancing the efficacy of mathematics OER for secondary school teachers and students. This objective contributed to ensuring the quality and student learning outcomes associated with mathematics OER. We examined the efficacy of our OER by assessing students’ immediate achievement and knowledge retention after 2 weeks.

Together, the research outcomes can inform future development of high-quality mathematics OER in a sustainable way. Based on the two objectives, the following research questions (RQ1 and RQ2a to RQ2c) are posed to guide the study.

• RQ1: How can a collaborative model involving pre-service and in-service teachers be designed to sustainably develop and refine mathematics OER?

• RQ2a: What are the perceptions of teachers and their suggestions for improvement regarding our OER created?

• RQ2b: What is the efficacy of our OER in terms of students’ immediate achievement and knowledge retention after 2 weeks?

• RQ2c: How do students perceive the use of our OER in terms of their understandability and impact on learning experiences?

Conceptual background

The conceptual background of this study encompasses three aspects: practical, methodological, and theoretical. First, we laid the groundwork for Objective 1 by reviewing the literature related to sustainable models for OER development. Second, we drew the methodology of formative evaluation in the study. After that, we addressed Objective 2 by delving into the theoretical foundation and tool which informed the development of mathematics OER.

Sustainable models of OER development

Tlili et al. (2023b) identified 10 sustainable models of OER development, with the internal funding model emerging as one of the most established. Doi et al. (2022) reported a noteworthy example regarding their university. Since the outbreak of the pandemic, their university leaders have decided to provide financial and technical support for OER development. This support enabled them to create OER to sustain students’ remote online learning. In light of students’ positive feedback, Doi et al. (2022) aimed to retain this e-learning component even after the pandemic. As Tlili et al. (2023b) noted, however, developers in the internal funding model generally face limitations due to the level of resource priority assigned to OER by institutional leadership.

Some OER developers adopted the community-based model, whereby “members of a community create materials for others to use” (Tlili et al., 2023b, p. 1428). This model usually involves interns, university students, and community members with minimal cost to sustain. For example, the interns of Boury et al. (2021) created microbiology OER as part of their internship requirements, which were modified due to the pandemic-led cancellation of their traditional internship programme. In a distinctive approach, Zhang et al. (2020) incorporated OER development into their family education course. Their teaching materials and students’ course assignments were published as OER. They reported that some students promptly shared these resources with their peers, demonstrating their sense of achievement and community involvement. To our best knowledge, however, few studies discuss the potential for pre-service teachers to serve as contributors to mathematics OER development.

Formative evaluation of OER

As defined by Tessmer (1993), formative evaluation is “a judgement of the strengths and weaknesses of instruction in its developing stages, for purposes of revising the instruction to improve its effectiveness and appeal” (p. 11) and “a problem-finding part of a design and product development process” (p. 12). In the field of educational technology and instruction design, formative evaluation is particularly significant in instructional design research aiming to improve and test feasibility (Moore et al., 2023). Based on Honebein and Reigeluth (2021), Table 1 summarises these two aims of formative evaluation and their implications for this study. Nevertheless, it is important to address a common misconception about formative evaluation. In research to test feasibility (as opposed to proving effectiveness), the focus is on evaluating the capabilities and limitations of resources, rather than comparing them with other counterparts (e.g., commercial materials). Therefore, the involvement of a control group is not necessarily required (e.g., Bakkum et al., 2022; Kanoksilapatham, 2021; Valle et al., 2022).

Table 1 The aims of formative evaluation and implementations for this study

Formative evaluation can play an important role in both ensuring and enhancing the quality of OER. Common techniques in formative evaluation during instructional design include expert reviews, one-to-one evaluations, and field testing (Moore et al., 2023). For example, Kanoksilapatham (2021) sought the input of three experts in English language teaching to ensure the quality of her OER. Their comments were used to validate the OER content. In health professions education, Bakkum et al. (2022) obtained feedback on their OER by conducting meetings with in-service teachers and integrated their suggestions into the development process. After the fourth meeting, no new suggestions were made, indicating teacher satisfaction with the resources. In addition to qualitative data-driven formative evaluation, Valle et al. (2022) evaluated OER-led learning outcomes in their course for citizen scientists. They generally scored over 85% in their assessments, indicating the efficacy of the OER produced. Notably, these quality assurance strategies entail minimal costs and seamlessly align with sustainable models of OER development.

Theoretical foundation and tool for mathematics OER development

Instead of merely relying on “common sense” or intuition, theories should be used to inform research and development in the field of educational technology. The central theory informing our OER development was Kolb’s (1984) Experiential Learning Theory, as this theory is suitable for creating interactive learning environments (Morris, 2020). The theory proposes a four-stage learning cycle: (1) concrete experience, (2) reflective observation, (3) abstract conceptualisation, and (4) active experimentation. To ensure the efficacy of mathematics OER, it is important to guide students through this learning process (Kolb, 1984).

Compared to the first stage of concrete experience, the later three stages are relatively less complicated to address (Morris, 2020). To facilitate students’ online learning, Yang et al. (2021) noted that instructional videos and learning sheets are common resources through which teachers can guide students’ observation (reflective observation) and help them create mathematics concepts (abstract conceptualisation). Teachers can then use online quizzes that prompt students to actively apply their knowledge to solve problems (active experimentation), allowing them to both practice and self-correct their mistakes (Barana et al., 2021). In their secondary school online mathematics classes, Hew and Lo (2020) found that the use of short-answer quizzes significantly improved students’ ability to solve novel problems, compared with just copying the teacher’s worked examples.

In the concrete experience stage, Morris (2020) emphasised the importance of physical interaction with learning objects. In an online learning environment, however, replicating this tangible engagement poses challenges. To enable a level of interaction similar to that in a physical classroom, teachers can use GeoGebra, an open-source dynamic mathematics software (https://www.geogebra.org/). According to Yohannes and Chen (2023), its abilities to manipulate variables using sliders and drag mathematical objects on the screen are the two most important dynamic features of GeoGebra. For example, Fig. 1 shows a GeoGebra applet for introducing equations of circles. Users can manipulate the variable r (the radius) using a slider and change the position of the centre A. Hence, they can experience and observe the changes in the equation of the circle. These features make GeoGebra an ideal tool for students to gain concrete experience during mathematical exploration.

Fig. 1

A sample GeoGebra applet for introducing equations of circles

Methods

Action research and procedure

The objectives of this study are to bring practical implications for the sustainable models of OER development (Objective 1) and to provide insights into future development of high-quality mathematics OER (Objective 2). The action research approach was therefore suitable because of its iterative and participatory nature to facilitate practical improvements (Creswell, 2012). Action research is a cyclic process of investigation. Our activities in its four main stages are described as follows.

• Planning: Setting the project objectives; defining the scope of research; preparing for the project activities; and identifying key research participants, including pre-service teachers and in-service teachers.

• Acting: Developing and refining the OER.

• Observing: Collecting and analysing feedback from in-service teachers; and examining the efficacy of OER using students’ tests and interview data (for the second action research cycle).

• Reflecting: Extracting valuable insights; identifying areas of improvement; and drawing lessons from both successful and unsuccessful aspects in the action research cycle.

After each cycle, the plan would be revised in the light of the experiences gained through observation and reflection (Creswell, 2012).

Figure 2 shows that this study comprised two action research cycles (AR Cycles 1 and 2). In each action research cycle, we adopted formative evaluation techniques to iteratively improve the quality of our OER. In AR Cycle 1, our research focused solely on improvement, involving in-service teachers in three iterations of review (a total of 10 months). Their evaluation and feedback were analysed and used to refine our OER. Upon completion of AR Cycle 1, we reflected and learnt from any effective and ineffective aspects. AR Cycle 2 was thus relatively smooth, and its duration (3 months with two iterations of review) was shorter than that of AR Cycle 1. Therefore, our research focus of AR Cycle 2 could expand to encompass not only improvement but also feasibility testing (Moore et al., 2023). Specifically, we invited student participants to help test our OER through learning assessments. To mimic students’ authentic online learning, they were given a one-day window to learn using our OER during after-school hours.

Fig. 2

Research procedure and data collection

The OER produced

The OER and some findings of AR Cycle 1 were reported in Lo et al. (2023). In that cycle, the OER were created by two final-year pre-service teachers, the third and fourth authors of Lo et al. (2023). The topic addressed was equations of straight lines. The present paper focuses on the OER in AR Cycle 2 with the mathematics topic of locus, which is one of the progression learning units of the topic addressed in AR Cycle 1. In this cycle, the OER were created by a final-year pre-service teacher, the second author of this article. The newly produced OER focused on two teaching objectives: (1) maintaining a fixed distance from a fixed point and (2) maintaining an equal distance from two given points (Curriculum Development Council, 2017). Our OER package included the following open-licensed materials in editable PDF format: (1) a summary of prerequisite knowledge (e.g., distance formula) with exercises; (2) an introduction to new concepts, accompanied by interactive GeoGebra applets; (3) learning notes and exercises; and (4) concept-checking questions with instant feedback (for the users who completed the questions via our website).

Take the GeoGebra applet for introducing the first teaching objective as an example (Fig. 3). The finalised applet and learning notes are available at https://www.geogebra.org/m/xpv3rn7u. We not only leveraged the important dynamic features of GeoGebra (i.e., manipulating mathematical objects and sliders; Yohannes & Chen, 2023) but also incorporated programming scripts into the applet. The slider with scripts controlled whether mathematical objects and instructions were shown or hidden, thereby guiding students’ reflective observation. Therefore, our GeoGebra OER were designed to engage students in both the concrete experience and reflective observation stages of Kolb’s (1984) learning cycle, facilitating a deeper understanding of mathematical concepts.

Fig. 3

Screenshot of a GeoGebra applet created in AR Cycle 2 with explanation

Research context

This study was conducted at The Education University of Hong Kong, the foremost teacher-training university in Hong Kong. We recognised that leveraging the potential of pre-service teachers could be a sustainable approach in our university. With each new cohort of students, the opportunity to create new OER annually emerged as a promising prospect. The university’s Student Empowerment Work Scheme funding aims to empower pre-service teachers through active ownership of project-based work and experiential learning opportunities. This initiative provides them with opportunities to assume responsibilities in planning and implementing projects—a philosophy aligned with the ethos of student-led OER development. Therefore, we adopted the internal funding model of OER development in AR Cycle 1. In AR Cycle 2, we transitioned to the community-based model. As to be explained in the Findings section, this shift not only attempted to capitalise on the innovative energy of our student community but also enhanced the sustainability of OER development.

Research participants

As we intended to elicit sophisticated feedback to improve our OER, experienced mathematics teachers (with at least 5 years of teaching experience) from different local schools were recruited using the purposive sampling method (Tongco, 2008). A total of 34 and 9 teachers were involved in AR Cycles 1 and 2, respectively. In AR Cycle 2, we further recruited student participants to test the use of our OER. The intervention was conducted at a local secondary school, and it involved 36 Grade 11 students who had not learnt locus problems (the topic of our OER). However, one student was excluded from our analysis because he obtained nearly a perfect score in his pre-test. Therefore, data from 35 students were analysed.

Data collection and analysis

This study involved five data sources. To address RQ1, researchers’ reflections on the development process were examined to identify the issues of the collaborative model for creating OER. To address RQ2a, teacher interviews and evaluation surveys were employed to gather their perceptions and suggestions for improving our OER. To address RQ2b, we measured students’ immediate learning outcomes and knowledge retention using achievement tests. To address RQ2c, we conducted student interviews to gain insight into their learning experiences with our OER.

Reflections (AR cycles 1 and 2)

In an action research study, reflection is central to generating actionable knowledge and developing practices (Coghlan, 2007; Leitch & Day, 2000). In the context of our study, we captured our observations, challenges encountered, and adjustments made throughout the OER development process in reflective journals. This reflective practice not only facilitated a deeper understanding of the collaborative dynamics and effectiveness of the process but also contributed to its improvement. Data from researchers’ reflections were analysed using thematic analysis, as suggested by Creswell (2012), which enabled the identification of key themes and insights that informed the development of a sustainable model for OER development.

Teacher interviews (AR cycles 1 and 2)

We collected teacher feedback through one-to-one evaluations (Moore et al., 2023) in a form of semi-structured interviews. Figure 2 shows that we conducted interviews (n = 7) only in Iteration 1.1 to collect teacher feedback in AR Cycle 1. After reflection, we relied more on interviews (Teachers A to C in both Iterations 2.1 and 2.2; Teachers D to I in Iteration 2.2) in AR Cycle 2. Richer qualitive comments were thus obtained for improving our OER, despite fewer teacher participants being involved.

Based on Bugler et al. (2017) and our reflection on AR Cycle 1, the interview topic guide was developed and covered five aspects: (1) accuracy and visual appeal, (2) alignment with standards and depth of knowledge, (3) ease of use and support, (4) engagement and ability to meet student needs, and (5) compatibility of content on multiple devices. Teachers were encouraged to provide detailed and specific comments on each aspect for improving our OER. Using the procedures proposed by Creswell (2012), thematic data analysis was conducted. Specifically, the interview data were coded by the first two authors. The identified codes were categorised according to the above five aspects. To enhance coding consistency, several representative quotes were identified to illustrate each constructed category. In the event of disagreements, the two authors resolved the discrepancies through discussion.

Evaluation surveys (AR cycles 1 and 2)

As shown in Fig. 2, we conducted evaluation surveys in both AR Cycles 1 and 2 (Iterations 1.2, 1.3, and 2.2). Adopted from Bugler et al. (2017), the original survey consisted of 15 close-ended items rated on a 5-point scale (ranging from 5 for “strongly agree” to 1 for “strongly disagree”) and one open-ended question for written comments. Upon completion of AR Cycle 1, we identified the need to include a new evaluation aspect, namely “Compatibility of content on multiple devices.” Therefore, three additional survey items were included in AR Cycle 2. Due to word limitations, we focus on the descriptive statistics of these new items, as presented in the Findings section.

Student achievement tests (AR cycle 2)

A 15-min pre-test, post-test immediately after the intervention, and a delayed post-test after 2 weeks were conducted in AR Cycle 2. The post-test measured the immediate impact of our OER on students’ learning outcomes, whereas the delayed post-test aimed to assess students’ knowledge retention over time. The questions in these tests, which were different but similar in terms of their scope and difficulty level, were designed based on Curriculum Development Council (2017). Each test comprised two test items based on our teaching objectives: (1) maintaining a fixed distance from a fixed point (9 marks) and (2) maintaining an equal distance from two given points (9 marks). The possible range of scores was from 0 to 18.

To analyse whether the scores between the three tests differed, a Friedman’s ANOVA test (a non-parametric test for comparing differences between several related datasets) was conducted (Field, 2009). This non-parametric test was used because the results of the Kolmogorov–Smirnov test discovered a violation of the normality assumption within the test data. When significant differences were found, multiple Wilcoxon signed–rank tests were run for pairwise comparisons. Applying Bonferroni correction in this post hoc procedure, the effects of the post hoc analysis were reported at a 0.05/3 = 0.0167 significance level (Field, 2009). We further calculated the effect size (r) using the formula of Field (2009) with the benchmarks of 0.1 (small), 0.3 (medium), and 0.5 (large).

Student interviews (AR cycle 2)

Semi-structured interviews (n = 6) were conducted to seek an understanding of student performance and learning experience. To obtain a wider range of perspectives, the students were divided into three performance groups based on their delayed post-test results. We then randomly selected two students from each group: 0 to 6 marks (Students L1 and L2), 7 to 12 marks (Students M1 and M2), and 13 to 18 marks (Students H1 and H2). The interview protocol was developed with two main questions: (1) “Could you elaborate on your performance on the tests?” and (2) “In what ways did the e-learning resources aid or hinder your understanding of the topic?” Similar to the teacher interviews, the first two authors analysed the student interview data using the procedures proposed by Creswell (2012).

Findings

A summary of findings from AR cycle 1

While some findings of AR Cycle 1 were reported in Lo et al. (2023), this section offers a summary and focuses on how our reflection on AR Cycle 1 informed the implementation of AR Cycle 2. First, our teacher participants confirmed a substantial demand for mathematics OER in teaching practice. The findings uncovered their key requests, including the need for more detailed guidelines, advanced materials, and interactive quizzes. For example, some teachers shared that students might struggle to observe mathematical properties without sufficient guidance. Therefore, they recommended that the OER should provide step-by-step descriptions and guiding questions for mathematical exploration. Notably, the 10-month duration of AR Cycle 1 suggested that our plan to develop OER for an entire topic was overly ambitious. Teacher reviews were both time-consuming and incomplete, potentially limiting their capacity to identify all issues in the OER produced. Consequently, the results of our evaluative survey indicated a non-significant improvement between Iterations 1.2 and 1.3 in terms of (1) accuracy and visual appeal, (2) alignment with standards and depth of knowledge, (3) ease of use and support, and (4) engagement and ability to meet student needs (Bugler et al., 2017).

RQ1: How can a collaborative model involving pre-service and in-service teachers be designed to sustainably develop and refine mathematics OER?

Our reflections on AR Cycle 1 and the corresponding improvements were organised in three major aspects essential to addressing RQ1. These aspects included (1) OER development model, (2) OER design, and (3) OER evaluation. For the OER development model, with support from an internal fund, our pre-service teachers were originally tasked with creating OER as part of their extracurricular activities. However, the development progress was slow due to their other commitments (e.g., coursework). To increase ownership and establish a more formalised approach, we attempted to incorporate the development into a final year project of a pre-service teacher in AR Cycle 2. Besides, developing OER for a whole topic was too ambitious, resulting in time-consuming and often incomplete reviews by teachers. Therefore, we refined our scope by focusing our effort on developing OER for two out of five teaching objectives in a topic.

For the OER design, it was difficult to use our GeoGebra applets as standalone resources because written instructions were only provided through worksheets. In our subsequent development, we embedded stepwise instructions in the GeoGebra applets for students to interact with (see Fig. 3). The teachers also expressed concerns about whether the OER could be displayed properly on mobile devices. Therefore, we conducted thorough testing to ensure our OER content was fully compatible across a range of devices and browsers.

For the OER evaluation, sole reliance on an evaluation survey led to teachers providing only concise written comments, which lacked the depth necessary to inform substantial improvement. Therefore, we expanded our feedback collection methods to include interviews with teachers, initially with three (Iteration 2.1) and subsequently with nine (Iteration 2.2), to gather more detailed and actionable insights. More importantly, the absence of student involvement in the initial testing phase of our OER left a gap in understanding their effectiveness from the student perspective. In this regard, AR Cycle 2 involved students in using our OER and evaluated their learning outcomes.

Findings from AR cycle 2

Drawing on the experience gained from AR Cycle 1, AR Cycle 2 proceeded smoothly. Overall, we only needed to conduct two iterations of review. In Iteration 2.1, we interviewed three in-service teachers for feedback on the OER. After revising the OER based on their comments, we interviewed the three teachers again to gauge their satisfaction with the updated resources, as well as six additional teachers for their comments. In Iteration 2.2, only minor revisions were required, making a third iteration of review unnecessary.

RQ2a: What are the perceptions of teachers and their suggestions for improvement regarding our OER created?

All of the teacher participants agreed that our OER would be useful in supporting student learning. In the words of Teacher B, “The design [of the GeoGebra applets] allows students to manipulate mathematical objects with their own hands.” During the interviews, the teachers were able to provide detailed and specific comments for improvement across four main aspects: (1) accuracy and visual appeal, (2) alignment with standards and depth of knowledge, (3) ease of use and support, and (4) engagement and ability to meet student needs (Bugler et al., 2017). In terms of the accuracy and visual appeal, Teacher A pointed out that “The statement ‘P passes through M’ is not a precise mathematical expression. As P is a moving point, it is the locus of P passing through other points.” Therefore, the statement has been amended to “The locus of P passes through M.” Representative examples of other three aspects are presented in Fig. 4.

Fig. 4

Representative examples of teacher suggestions and corresponding follow-up actions

In AR Cycle 2, we further considered the compatibility of content on multiple devices. In Iteration 2.1, the teachers generally expressed that the compatibility of our GeoGebra applets was not satisfactory. Teacher B pointed out that “the website is smooth on iPad. However, other devices, such as mobile phones, might face compatibility issues. The size of the GeoGebra applets is not adjustable.” To address this problem, we adjusted the dimensions of all applets from 700 × 600 to 700 × 400, as shown in Fig. 5. The new dimensions are closer to the screen of a cell phone. We also changed the types of concept-checking questions from fill-in-the-blank to multiple-choice questions to make it easier for student input. In Iteration 2.2, all of the teacher participants supported these improvements. No further revisions were requested, which was consistent with their satisfactory results reported in the evaluation survey (Table 2). For example, almost all of the teachers agreed or strongly agreed with the statement, “The resources run smoothly across devices and platforms.”

Fig. 5

Example of improving the compatibility of content on multiple devices

Table 2 Teachers’ (n = 7; response rate = 77.8%) evaluation survey results of the compatibility of content on multiple devices

RQ2b: What is the efficacy of our OER in terms of students’ immediate achievement and knowledge retention after 2 weeks?

Figure 6 shows the boxplot of the students’ (n = 35) scores across the pre-test, post-test, and 2-week delayed post-test. Friedman’s ANOVA indicated significant differences between the three tests (χ²(2) = 48.97, p < 0.001). Wilcoxon signed–rank tests were used for pairwise comparison. The results indicate that compared with the pre-test (Mdn = 2.00), the students scored significantly higher on the post-test (Mdn = 10.00, p < 0.001) and delayed post-test (Mdn = 11.00, p < 0.001), both with a large effect size of r = 0.62 and r = 0.61, respectively. However, the difference between the post-test and delayed post-test scores was not significant (p = 0.75).

Fig. 6

Boxplot of the students’ (n = 35) scores by test

Notably, Table 3 shows that student performance on Test items 1 and 2 were different. For Test item 1, more than 60% of the students attained 7 to 9 marks in the post-test and delayed post-test. This result indicates that they generally mastered the concept of maintaining a fixed distance from a fixed point. For Test item 2, however, nearly half of the students scored no more than 3 out of 9 marks on both post-tests. In other words, they still required additional support in mastering the concept of maintaining an equal distance from two given points.

Table 3 Number of students (n = 35) in different score ranges by test item

RQ2c: How do students perceive the use of our OER in terms of their understandability and impact on learning experiences?

All of the student interviewees (n = 6) confirmed the effectiveness of using our OER to learn, especially the GeoGebra applets. As Student H1 expressed, “The software makes it clear the locus to me, which enables me to have a better understanding.” They also stated that the clear instructions with solutions in the notes facilitated their learning. One student appreciated the design of our concept-checking questions: “These [questions] consolidated my concepts right after study. I love the way that they provided immediate feedback for me to check my understanding” (Student M1). Overall, four out of six students appreciated this kind of online independent study, which allowed them to learn at their own pace.

In terms of their performance, students in the three achievement groups provided explanations for their test results. Student H1 performed the best in his delayed post-test. He shared that after taking the post-test, he revisited the OER to review something that he did not understand before. The OER allowed him to adjust his learning progress. If the concept was difficult, he would spend more time doing drills on it. Student M1 achieved the same score in both the post-test and delayed post-test. He attributed his ability to revisit the GeoGebra applets, which provided him with a deeper understanding of the locus. However, Student L2 complained that some of the content was difficult to learn without teacher assistance.

Discussion

As sustainability and quality are major issues of OER development (Perifanou & Economides, 2023; Sousa et al., 2023; Tlili et al., 2023a), this action research study attempted to leverage the skills of pre-service teachers to develop mathematics OER for secondary school teachers and students. Our broader objectives are (1) to develop a sustainable model for creating OER and (2) to propose strategies for enhancing the efficacy of mathematics OER. Hence, our findings are discussed with reference to these two objectives.

Towards a sustainable model of OER development with pre-service and in-service teachers

Our conceptual model of OER development is presented in Fig. 7, which highlights the involvement of pre-service and in-service teachers as valuable contributors to the process. Feedback from the teacher participants confirmed the usefulness of the OER created by them. However, we encountered challenges in AR Cycle 1 due to the pre-service teachers’ competing commitments, which hindered the progress of the project. To address this, we incorporated OER development into a pre-service teacher’s final year project (as an OER-based project), similar to what were done in Boury et al. (2021) and Zhang et al. (2020). Due to the shift from being consumers to active producers of educational resources, student motivation and commitment were increased (Zhang et al., 2020).

Fig. 7

Conceptual model of sustainable OER development with pre-service and in-service teachers

Although the use of formative evaluation techniques in OER development can be effective (Moore et al., 2023), it is important to prioritise quality over quantity. First, limiting the scope of resources created each time can ensure smooth development progress and save time for teachers who review the resources. Second, our experience indicated that involving a few teachers in one-to-one evaluation is more meaningful than using surveys for collecting feedback, as written comments might not be sufficient for them to provide detailed suggestions for improvement. OER developers can prompt teacher feedback based on the five aspects of OER evaluation (Bugler et al., 2017) presented in Fig. 7. Similar to Kanoksilapatham (2021), we first invited three experienced teachers to provide feedback in AR Cycle 2. As in Bakkum et al. (2022), we then scheduled another iteration of review to collect teacher feedback until no new suggestions were made. Additional teachers were involved to gain broader perspectives. By focusing on quality over quantity, we were able to finalise our OER in two iterations of review in AR Cycle 2 without overwhelming our teacher participants.

Strategies for enhancing the efficacy of mathematics OER

The mobile compatibility of OER is crucial, as several studies (e.g., Kanoksilapatham, 2021; Zhang et al., 2020) have found that some students use their mobile devices to access online resources. In AR Cycle 1, the teachers expressed concern about whether our OER could function effectively on mobile devices, and this concern was reiterated by the teachers involved in AR Cycle 2. As a result, we made two major revisions: (1) adjusting the dimensions of the GeoGebra applets to 700 × 400, and (2) ensuring that the concept-checking questions were mobile-friendly by using multiple-choice questions, although short-answer quizzes may have a better effect on students’ mathematics learning (Hew & Lo, 2020). Practically, such a use of multiple-choice questions allows students to easily select and check answers through mobile devices (Wilson, 2017).

In this study, GeoGebra applets were the main resources developed to facilitate students’ mathematical exploration. They could gain concrete experience to support subsequent stages of Kolb’s (1984) experiential learning cycle. Teachers in both action research cycles confirmed that our OER were useful in their teaching practice, indicating that the resources can serve as a viable alternative to commercial materials (Hilton, 2020). In AR Cycle 2, our student participants also expressed that the OER helped their understanding and retention of mathematical concepts. These findings are consistent with the study of Yohannes and Chen (2023), who found that using GeoGebra had positive effects on students’ learning performance. Specifically, our students performed well on questions related to maintaining a fixed distance from a fixed point, with a median score of 8 out of 9 marks in the two post-tests (see Table 3).

However, better strategies for enhancing the effectiveness of mathematics OER still need to be explored, particularly in terms of advanced topics. More detailed examination of the students’ test scores showed that they did not perform well on the more advanced test item of maintaining an equal distance from two given points (see Table 3). Although this is consistent with students’ general performance in Hong Kong (e.g., HKEAA, 2021), it highlights their difficulties in mastering knowledge during online independent study without teacher assistance and the limitation of our OER. In this regard, we propose two possible strategies based on recent advancements in educational technology:

• 1. Develop open source chatbots to support students’ online learning. These chatbots can be programmed to provide answers to frequently asked questions related to specific learning materials (Hwang & Chang, 2023).

• 2. Create mathematical objects in open source metaverse applications for students to experience. Objects in the metaverse may provide better visualisation of loci and other mathematics concepts (Topraklıkoğlu & Öztürk, 2023).

Limitations and recommendations for future research

Two major limitations of this study must be acknowledged. First, the study was conducted at one education university and only involved two mathematics topics. Further studies are required to investigate a wider range of topics in different contexts. Such studies can provide additional insights to enrich our proposed sustainable model and strategies of OER development.

Second, while we had examined student achievement and knowledge retention, we did not examine how teachers use our OER to support their teaching practice. However, teacher unpreparedness and lack of prior experience can affect the successful implementation of OER (Stracke et al., 2022). Therefore, further research is necessary to explore the ways in which teachers use OER and the challenges they face in the process. This would provide more information for improving OER design and inform the development of training programmes on OER adoption.

Conclusion

This study responded to call of UNESCO (2020), addressing the education of students who may have less access to commercial materials in the post-pandemic world. Using the action research approach, we established a sustainable model of OER development with pre-service teachers in mathematics education. The model further involved in-service teachers in formative evaluation to ensure the quality of the OER. Although the use of the OER allowed students to interact with virtual mathematical objects and resulted in a significant learning gain, their effectiveness in helping students learn more advanced concepts was limited. Future research is required to explore the integration of more advanced educational technologies to enhance the efficacy of OER.