Co-created augmented reality app and its impact on the effectiveness of learning a foreign language and on cultural knowledge

Abstract

This paper addresses the limited exploration of the impact of virtual and augmented reality on the learning process, particularly within the context of foreign language learning among young learners. Motivated by this research gap, the EULALIA, conducted from 2019 to 2023 (2019-1-IT02-KA203-063228), sought to evaluate the efficacy of integrating technology in language education. Specifically, the study investigates the influence of virtual and augmented reality on foreign language learning, examining its effects on subjectively assessed cultural knowledge, and workflow during classes, as perceived by educators. Involving 239 learners, the study employed a pre/post survey methodology, wherein participants provided feedback before and after working with the EULALIA application or traditional learning methods. The quantitative analysis of survey data was complemented by semi-structured interviews with participating teachers, providing additional insights. The results indicate a statistically significant increase in cultural knowledge among the group utilizing augmented reality. The findings suggest a positive impact on students’ learning progression, and workflow during classes. This research contributes valuable insights into the potential of integrating virtual and augmented reality in language education, offering a foundation for future pedagogical advancements.

Introduction

Cross-cultural communication is an inevitable part of human functioning in the globalised world, in which people not only migrate from one place to another but are also supposed to communicate in a culturally driven context (Paige et al., 2000). The need for rapid adjustment to new conditions has led to attempts to incorporate elements of cultural awareness training into curricula across various disciplines. Although the time spent in a foreign country leads to a better understanding of cultural differences, in many cases, the first contact with a foreign culture becomes a real challenge. Moreover, according to Gao et al. (2021), the balanced development of cultural awareness when studying in a foreign country is hindered by a lack of time and opportunities. Learning a foreign culture should focus on the situations and contexts that a person will most probably encounter in a foreign country. Modern technologies offer a solution to time limitations by alleviating stress through a self-regulated learning path and placing the learning in a ready-to-discover virtual world.

The designers of the EULALIA project¹ created a virtual reality using real-life scenarios and multisensorial methodologies, applying Tangible User Interfaces (TUIs) and mobile learning to check the potential of a new solution in the acquisition of cultural content.

The paper explores the effectiveness of using digital reality in developing cultural awareness from the student’s perspective. The aim of the study is to investigate the influence of virtual and augmented reality on foreign language learning, and in particular to examine its effects on perceived cultural knowledge, and workflow in class, observed by teachers.

Literature overview

The presentation and discussion of the study results are preceded by a short theoretical introduction, in which the significance of learning cultural aspects and the role of virtual reality in language teaching and learning are briefly explained, and the main principles and functioning of the EULALIA application are outlined.

Cultural learning

Due to its highly complex nature, language and the target community’s culture are inextricably intertwined. Moreover, the development of (inter)cultural communicative competence (ICC) “supports the successful and sufficient communication in different languages” (Byram, 1997: 22). Byram (1997) distinguished five closely interconnected elements supporting the development of ICC:

• knowledge,

• interpreting and relating,

• discovery and interaction,

• critical cultural awareness,

• curiosity and openness.

In spite of criticism of the model, due to its perceived limitations in an era of globalisation (Hoff, 2014), the model includes crucial elements needed for the development of cultural awareness, such as knowledge about the culture related to diverse fields, such as geography, cuisine, art, and the like, stimulating learners’ curiosity about life-based cultural situations or enhancing openness and readiness for interaction. These aspects can be perfectly incorporated into virtual reality in order to open up new learning opportunities. With a changing approach to digitisation in the world and the classroom, students should be properly supported to help them engage in learning (Henrie et al., 2015).

Sein (2022) investigated the beliefs of learners about learning culture and proved that it is as important as learning foreign languages, and should be extensively incorporated into the curriculum, “[t]he question at the time being is not whether to teach culture to students of second or foreign languages or not, but rather the valid question should be how to best teach culture to students in second or foreign language classrooms” (Sein, 2022: 8). According to the results of Sein’s (2022) study, as much as 48% of the students surveyed confirmed that they regularly download additional online culture-loaded materials to discuss in the classroom and extend their knowledge. Liu et al. (2022) revealed that international students recognise cultural differences in their new environment (studying abroad). The authors also mention the significance of increasing knowledge about the customs and culture of the destination country, such as food, festivals, and clothing, for fostering students’ motivation. In the motivation studies, age difference is also mentioned as an important factor. A GALLUP study (2018) informs that the highest level of engagement can be noticed in the last classes of primary school, but that after that period there is a steady decline, which can be explained by the changing needs of older students and the lack of suitable means to encourage student engagement.

The recent outbreak of the COVID-19 pandemic revealed the new face of engagement, namely that the online environment, previously seen as a motivator stimulating the attractiveness of the teaching and learning process, contributed to enormous challenges in developing student involvement over a longer period of time (Kilickaya et al., 2022). Researchers also point out that involvement does not affect all students in the same way, and some experiences might have a negative impact or impede students’ development (Mainella, 2017). It is therefore important to use a variety of approaches in the learning process, and to analyse their effectiveness.

Virtual environment in the learning process

Milgram and Kishino’s (1994) reality-virtuality (RV) continuum represent a milestone in the application of the paradigm in the fields of engineering and computer science. Importantly, the authors attempted to propose a taxonomy in a sector where setting any boundaries is a challenge (Koleva et al., 1999). Milgram and Kishino (1994) started with three main concepts: the extent of world knowledge, reproduction fidelity, and the extent of presence metaphor. In their simplified version of the RV continuum, they applied the concepts of real environment and virtual environment, which represent the two opposite ends of the continuum. Once we include real elements that interact with digital content, we are moving along this continuum, and all of the elements in between fall into the scope of mixed reality (MR) (see Fig. 1).

Fig. 1

Virtuality continuum theorised by Milgram and Kishino (1994)

This reality-virtuality continuum has been updated and revised over time (see Skarbez et al., 2021; Speiginer & Maclntyre, 2018), also because, in recent years, new technological devices have emerged, necessitating profound reflection on the concepts.

Augmented reality (AR) represents a field close to the real environment (Milgram et al., 1994) since it allows users to enhance reality using digital assets. AR systems are able to improve real-world objects with digital and virtual components, including superimposition. An AR environment has the ability to “supplement […] reality, rather than completely replacing it” (Azuma, 1997). There are two main types of AR:

• Marker-based AR.

• Marker-less AR.

The types of marker-less AR differ based on their application in the real world (Aggarwal & Singhal, 2019). Thus, all systems where additional visual elements are overlaid onto a physical object to allow human interaction with the projected images could be defined as Projection Based Augmented reality. If, on the other hand, a visual image is only added to the system to improve the original view, this would be defined as Superimposition Based Augmented Reality.

Augmented Reality gained currency once mobile technology improved. With the advent of NFC-enabled mobile devices, it became possible to design systems where the user could hover a mobile device with sensors over a physical object in order to access Augmented Reality features for interacting with the real world. A subcategory of AR is the application of Tangible User Interfaces (TUI) (Ishii & Ullmer, 1997), where the interface is physical and real (an object in an environment that could guide a digital interface by its internal features). The real object, namely the Tangible User Interface, contains digital information using different technologies (QR codes, NFC antennas, shape and contour characteristics, weight detection, etc.). The physical object represents a vehicle for the digital information that can be easily transduced in a digital environment.

All these technologies have been applied in a variety of fields, and the educational setting has also benefited from these tools (see Bower et al., 2014; Elmqaddem, 2019; López-Belmonte et al., 2023; Wu et al., 2013) in the context of the Technology Enhanced Learning (TEL). In this context, the authors describe interactive maps as physical tools able to hold information and provide digital (audio and/or visual) feedback to the learner. An example is FingerTrips (Palaigeorgiou et al., 2018), which is a continuous shape display that supports geography teaching by involving the sense of touch to collect information and allowing learners to play with cards. Another example is the SandScape app designed by Ishii et al. (2004), which enables interaction with sand that is illuminated, providing an environment for simulation of the Geographic Information System. Other similar solutions are project-based, as in the case of a prototype of a multisensory augmented reality map used to support visually impaired students (Albouys-Perrois et al., 2018).

The potential of these interfaces lies in allowing the student to interact with a real environment enriched with digital information, and increased interaction with the platform means gathering more data thanks to a multisensory approach (Ponticorvo et al., 2019). Augmented reality applications coupled with gamification aspects are a well-known strategy in order to increase psychological variables such as motivation, engagement, and curiosity of students. These benefits are extended also to learning outcomes and academic performance (Garzón et al., 2019; Lampropoulos et al., 2022) with a direct effect on the students’ performance (Chang et al., 2022). In particular, a specific positive effect is related to language learning promoting students’ positive responses for studies that have a long-term intervention (Chang et al., 2022). This impact is also present when the AR is applied for the support of individuals with special needs, where it can improve individual learning skills (Baragash et al., 2020; Yenioglu et al., 2023).

Augmented reality tools have been shown to have a positive effect on spatial thinking (Carbonell Carrera & Bermejo Asensio, 2017a, 2017b; Majeed & Al Rikabi, 2022) compared with traditional materials. Studies have focused mainly on knowledge improvement in geography and related fields, and have shown that the use of AR reinforces the ability to make connections between objects, to categorise and to locate objects in space (McLaughlin & Bailey, 2023).

In the case of the present article, the authors present a tool that does not focus on geographical competence but aims to improve cultural and language knowledge of foreign students in higher education settings.

An overview of the EULALIA project

EULALIA was a project implemented under the Erasmus+ Strategic Partnership² programme. The project partners included the University of Naples Federico II, from Italy (UNINA), the University of Alicante, from Spain (UA), Smarted S.R.L, from Italy (Smarted), the European University Foundation-Campus Europae, from Luxembourg (EUF), the University of Malta (UM), and Adam Mickiewicz University in Poznań, from Poland (AMU). The project was conducted in the academic years 2019–2022 (total duration: 30 months).

The primary aim of the project was to improve and unify the teaching strategies of university language centres for Erasmus students in the project’s member countries. This was achieved through the development of a learning and teaching tool based on mobile learning and game-based learning.³ The second objective was to observe the potential impact of the Tangible User Interfaces (TUIs) setting that enables the learner to interact with cards, maps, blocks, or objects from everyday life in the learning process.⁴

The project plan included the preparation of mobile applications for Erasmus students to help them develop knowledge about the cultural, social, and linguistic aspects of their place of stay—the foreign country they were travelling to as part of the Erasmus programme. To increase the effectiveness of the project and, at the same time, prepare a tool that could best meet the existing needs and expectations (digitisation, gamification, easy access via mobile devices), the participating students were involved in the creation process as part of the teaching and learning approach.

Learning tool: EULALIA app

The main variable underlying the research we conducted is the EULALIA application, which supported the teaching of a foreign culture and language in the test groups. The app, made as part of the EULALIA project (Brunetti et al., 2021), is an augmented reality (AR) tool compatible with smartphones equipped with Android operating systems and NFC antennas. Different national groups in each of the four countries developed maps as part of OER, including information about specific aspects of the town, region, or country where the university is located (Mazzuccato & Kic-Drgas, 2021). The maps were enhanced with NFC tags (round, 15 × 15 mm, ISO 15693, ISO 18000-3 Mode 1, 13,56 MHz). These are passive antennas (Fig. 2a) placed on the back of the maps in “hot spots” (Fig. 2b), where additional information would be useful to achieve the learning aims (i.e., the location of the central station of the city, places of historical interest, places to eat, etc.).

Fig. 2

NFC tags (left) and NFC santennas (right)

The user interacts with a specific map by downloading the related app on their smartphone, available for free through the Google Play application store. In this way, the student is able to interact with the EULALIA augmented reality system by browsing the map (Fig. 3). The learning exercise, designed in order to improve cultural knowledge of the host city or region, outputs aural feedback, and the student is asked to find the right place on the map by tapping their phone on the hot spots. The hot spots are highlighted with blue pins or transparent clips. Once the smartphone is over a hot spot, the antenna detects the NFC tag and gives aural feedback. If the answer is correct, the player can proceed to the next question. The language difficulty is balanced so that learners at most levels can use the app, some applications are developed for B1 or below, others for B2 or above. The games allow the learners to browse the map and, at the same time, learn new words and obtain new information about the place of stay.

Fig. 3

Student’s interaction with EULALIA

The pictures show two students who are interacting with the EULALIA map by tapping their phones on it. On the back of the map, there are the NFC tags that are detected by the embedded antenna. If the student taps the right spot, they get positive feedback from the device. If not, then they are asked to try again.

Research design

In our study, we used the EULALIA app to teach the language of the country in which the participants were doing their international exchanges (e.g. through the Erasmus programme). In particular, we wanted to ensure that in linguistically diverse countries (such as Malta or Spain) the classes covered a regional or native language. Accordingly, tests were carried out within Italian, Maltese, Catalan, and Polish as a foreign language classes. In our study, we focused on assessing the effectiveness of learning elements of a foreign language country’s culture with and without augmented reality. Moreover, after the testing phase of the app in the classroom, we asked teachers about their perceptions of working with the app in their foreign language classes in order to garner in-depth and contextual insights beyond the scope of the quantitative survey. Specifically, the project partners sought to elucidate the perspective of teaching staff regarding the incorporation of Augmented Reality (AR) in foreign language classes. This involved an assessment of the educators’ experiences with the application during class sessions, an exploration of its influence on the overall class dynamics. The interviews aimed to provide a comprehensive understanding of the instructional landscape and the nuanced effects of integrating AR technology into language education, as perceived by the teaching faculty.

The authors posed the following research questions:

RQ 1:

Does the incorporation of virtual and augmented reality within foreign language classrooms yield a statistically significant increase in the (subjectively assessed) knowledge of the culture associated with the foreign language country?

RQ 2:

How does, from a teacher’s perspective, the use of augmented reality affect the atmosphere and workflow in the classroom?

To gather data from participants in partner countries of the project, both pre-survey and post-survey assessments were conducted online. The surveys focused on self-assessments of participants’ knowledge regarding the culture of the host country. Additionally, in order to augment the information obtained from the surveys and to gain further understanding of the application’s utilization and its impact on student learning, online interviews were conducted with participating lecturers. The interviews provided supplementary insights into the usage patterns of the application and its effects on the educational experience of students.

The study comprised six phases: (1) an initial stage involving a needs analysis targeting international students, (2) the inaugural workshop phase dedicated to identifying interested participant groups, during which teachers (and some students) were acquainted with the application’s functionality and the principles of scenario development, (3) the subsequent workshop phase focused on the co-creation of learning scenarios, (4) the technical implementation stage of these scenarios, (5) a teaching phase integrating the application, and (6) a phase dedicated to data collection and analysis. It is worth noting that the data collection and analysis stage commenced partly before the teaching phase, incorporating a quantitative pre-test, and concluded subsequent to the teaching phase, culminating in a post-test assessment (Fig. 4).

Fig. 4

Phases of the EULALIA project

Not all phases can be comprehensively explained in this paper. Consequently, stages 1–5 will only be delineated below. Data collected in the 6th stage will be discussed in detail in Chapter 3.2.

Initial phases of the project

The aim of the EULALIA project was to help students arriving in a foreign country find their way into a new reality, a new culture, and new customs. The development of the digital course outlines was preceded by a survey among exchange students (mostly Erasmus). The survey investigated the needs of the respondents arising from their arrival in a foreign country and the difficulties they encountered during their stay abroad. Parallel to the survey, short interviews were conducted with teachers working at the university language centres, who also pointed out the greatest challenges—cultural, logistical, or communicative—faced by foreign students. The needs and difficulties identified by the largest group of respondents and interviewees served as a starting point for the project consortium to prepare the teaching and learning scenarios. The most frequently mentioned needs revolved around the following thematic areas: local cuisine, geography, history, literature, sport, values, and differences in pragmatics.

Having identified the main thematic groups of scenarios, the project consortium proceeded with two co-learning phases of the project, which involved training language centre teachers and exchange students to create the scenarios themselves. Students and teachers were familiarised with the principles of the project and participated in a training session showing how the EULALIA app functions and how the OER should be developed. The workshops helped them understand the core idea of the use of the application. During and after the workshops, students and teachers were creating their own scenarios. After designing a total of 31 scenarios, Smarted srl digitized them and made them available for download on Android phones through the Google Play platform. This enabled the commencement of the foreign language teaching process through the application.

Testing phase

Between October and December 2021, the scenarios were tested with a group of students in Poland, Italy, Malta, and Spain (the test group, or the EULALIA group). The EULALIA group worked with the app for Android smartphones and with maps with interactive hot spots.⁵ The same topics were discussed in traditional classroom settings without the enhancement of the AR system, namely without the use of the EULALIA app. This group represents the control sample (the reference group). The same traditional supplementary materials (handouts, exercises) were used in both the test and reference groups.

The students were asked to complete two surveys called the pre-survey and post-survey, which were disseminated during the first weeks (pre-survey) and at the end of the language courses (post-survey), respectively.

The pre-survey (see “Appendix 1”) comprised 23 questions. The first six questions related to the respondent’s personal data, native language, status as an Erasmus student, and level of understanding of the target foreign language. Question seven aimed at the respondent’s self-assessment of their knowledge of the host city. Here, the participant could rate his or her knowledge on a scale from 1 (basic) to 10 (advanced). Questions 8–18 were structured as closed inquiries, enabling respondents to assess their expectations through a five-point scale ranging from “strongly agree” to “strongly disagree”. Questions 8–14 prompted participants to evaluate the extent to which they anticipated receiving instruction on various topics, encompassing local cuisine, geography, history, literature, sport, values, and differences in pragmatics, specifically politeness. Questions 15–18 referred to expectations concerning knowledge of the university’s history, finding your way around campus, navigation in the city, and a famous person from the area. Question 19 asked about the preferred learning style (constructivist, collaborative, enquiry-based, integrative) and question 20 allowed the learner to justify their choice. Questions 21–23 asked about digital skills, including in particular the need to implement digital skills activities in foreign language classes, a self-assessment of digital skills on a 10-point scale, and expectations for the development of digital skills after the project.

The post-survey (see “Appendix 2”) repeated the questions from the initial survey and verified the expectations of the respondents, showing how they actually felt after taking part in the project using the app. The surveys were made available via Google Forms.

The teachers involved in the study participated in single semi-structured interviews aimed at investigate the perspective of how the use of augmented reality affects the atmosphere and flow in the classroom.

Participants

Regarding RQ1, a total of 239 students took part in both surveys. The survey sample consisted of Erasmus students in the country concerned. Invitations to take part in the survey were sent out to all students in the university’s database at the time, and students took part voluntarily. Only 20–30% of the students responded, which was probably connected with additional time to spent at the university and additional classes they were supposed to participate in. In addition, it is worth mentioning that the number of students participating in the post-survey was reduced, as not all students were able to participate due to illness or decided to return to their country.

Although as many as 328 students participated in the pre-survey, the number of participants in the post-survey was 27% lower, for various reasons. Firstly, this was due to the rather unstable educational situation related to the COVID-19 pandemic. Some international students could not continue their studies. As it was not possible to identify those who took part in both the pre- and post-survey, most of the results presented will be given as averages (AVG), percentages and not absolute values.

A total of 117 students were assigned to the EULALIA app testing group and 122 students to the reference group. The individual distribution of the participants in the study was as follows (Table 1).

Table 1 Participants in the test group (TG) and the reference group (RG) in the pre-and post-survey by country

It is noticeable that there was a significantly greater attrition of survey participants in the test group than in the reference group.

The course participants had very different levels of knowledge of the target language: from real beginners up to C2. The distribution and number of participants by country of study and language proficiency (Italian, Maltese, Catalan, Polish) are shown in Table 2.

Table 2 Participants in the test group (TG) and reference group (RG) by language proficiency level and country

The participants in the study were all international students, including both Erasmus- and non-Erasmus students.

In the case of RQ2, lecturers were involved in the study performing semi-structured interviews. The sample group consisted of 5 lecturers (one from each member country) and the interview was conducted with lecturers teaching Polish, Spanish, Italian, Maltese as a foreign language using the EULALIA app.

Survey results (RQ 1): impact of the EULALIA application on the enhancement of knowledge about the host country

One of the main points of the study was to investigate whether working on and with the application helps to gain knowledge (cultural, topographical, geographical, historical) about the host country of the exchange program. The study was based on a self-assessment. The participants were asked to rate their knowledge on a 10-point Likert scale, from 1 (very little knowledge of the country) to 10 (outstanding knowledge of the country).

Test group

In the pre-test phase, the mean knowledge score concerning the host country was 4.83 for all participants in the test group. Participants studying in Spain had the highest knowledge, and those studying in Italy the lowest. The data on the average knowledge of the host country were supplemented with information on the standard deviation (SD). The results were closest to the average for students from Poland, whereas the largest deviation was recorded for students from Malta. With regard to language level, the deviation is approximately 2 and is comparable for all levels (Table 3).

Table 3 Initial self-assessment of students in the test group of their knowledge of the host country (pre-survey)

Interestingly, the level of knowledge about the country did not correlate with the level of linguistic knowledge. Students at B2 level claim to have the highest level of knowledge of the country, with the lowest self-assessment visible at C2 level, however, this result can hardly be considered representative, due to the low number of respondents in this group.

In the post-survey students mostly indicated an increase in their knowledge compared to the beginning of the courses (Table 4).

Table 4 Final self-assessment in the test group regarding the knowledge of the host country (post-survey)

Comparing the initial results with the results obtained after working with the EULALIA application, an increase in self-reported knowledge of the host country was seen in each study group. At the same time, the standard deviation decreased for students from Malta and Poland, while it slightly increased among students from Italy and Spain. Compared to the pre-survey results, the overall level of knowledge of the host country increased by 17% from 4.83 to 5.84 (Fig. 5).

Fig. 5

Difference in self-assessment of test group participants for individual countries and in total

For an even better illustration of the results of the self-reported knowledge of the host country, the pre-survey and post-survey results are juxtaposed once again below, this time, however, the median (MED) response is shown (Table 5).

Table 5 Difference in self-assessment of test group participants expressed as a median

Thus, analysing the aggregate data it can be seen that the median response did not increase only in Spain. Among students at the University of Naples the median increased by 1, among students from Poznań by 1.5 and among students from Malta by 2. Overall, the median increased by 1.

A t-test was performed for the test group, who used the EULALIA app. The difference between the pre-survey results and post-survey results is statistically significant (see Table 6) with 99% confidence, in particular, the group using EULALIA showed an increase in the perceived knowledge of cultural sites.

Table 6 T-test for the test group, before and after the use of the EULALIA application (independent samples T-test)

A statistically significant difference for the use of the EULALIA app is also retained when the analysis is performed excluding the data from the Italian sample, whose results differed most from the data from the other countries, and where the dropout could cause misleading results.

Reference group

From the very beginning, the respondents in the reference group rated their knowledge higher (by 1.2) than in the test group.⁶ The standard deviation, however, is also higher, with an especially high value of 2.73 for the whole group studying at AMU in Poznań (Table 7).

Table 7 Initial self-assessment in the reference group regarding the knowledge of the host country (pre-survey)

Similarly as in the test group, an increase in knowledge of the host country can be observed in post-survey in almost every country except for Poland (Table 8 and Fig. 6). A very high increase of almost 1.5 points was recorded by the students at the University of Naples. However, the total increase in knowledge is smaller (0.73 points) than in the test group, which is an increase of around 11% (Fig. 6).

Table 8 Final self-assessment in the reference group regarding the knowledge of the host country (Post survey)

Fig. 6

Difference in self-assessment of reference group participants for individual countries and in total

The t-test was applied for the reference group in the same way as for the test group (Table 9). In this case, the difference is significant: the mean of the pre-test is less than the average results obtained in the post-test with the confidence interval 95%, being p = 0.003.

Table 9 T-test for the reference group (independent samples T-Test)

When the Italian sample is excluded from the dataset (see Table 10), the difference between the pre-test and post-test is not statistically significant (p = 0.262).

Table 10 T-test for the reference group excluding the data from Italy (independent samples T-test)

To summarise the results, the data are shown as boxplots, both for the complete dataset (Fig. 7a) and excluding the Italian sample (Fig. 7b). The boxplots highlight the increase in self-assessed cultural knowledge in the test group after the course using the EULALIA app.

Fig. 7

On the left a are shown the results for the whole sample, on the right b the results with the Italian data excluded

Interviews with lecturers (RQ 2): impact of AR on atmosphere and workflow during classes

The semi-structured interviews with lecturers of Italian, Maltese, Catalan, and Polish as a foreign language, lasting from 20 to 35 min, were conducted in October and December 2021 and June 2022 to supplement the results from the questionnaires.

In the following part of the paper, we will focus on three interviews conducted by the authors of this paper with lecturers teaching Polish as a foreign language using the EULALIA app. All interviews were conducted in Polish to give teachers more freedom in expressing themselves, the translations have been prepared by the authors. The interviews are treated more as a supplement to the questionnaire results and shed light on the students’ involvement in lesson activities and learning new cultural facts. The information gathered in the interviews relates to co-creation in teaching, its influence on the results and benefits for teachers of working with the EULALIA app (Tables 11, 12 and 13).

Table 11 The average and median knowledge increase/decrease in the test group for each language level

Table 12 The average and median knowledge increase/decrease in the reference group for each language level

Table 13 Improvement of digital skills in the test group

The teachers participating in the interviews admitted that the use of the EULALIA application helped them change the way they conducted their classes:

EULALIA has made me change the structure of my classes, open up to new digital possibilities, and also start to understand the students better. I1⁷

The application helped me to organise and design my class, implementing linguistic-cultural paths in order to develop cooperation. I2

The participants of the interviews underlined the value of collaboration in the project. The co-creation approach significantly supported the students’ creativity and the possibility of learning one from another. Students in the EULALIA group were more engaged and active, since they felt a part of the project. The involvement also contributed to the increased willingness to take responsibility for one’s own learning process.

Perhaps the most interesting part of the project was the collaboration, the opportunity to exchange experiences and the active involvement of both learners and teachers. I1

Working with the application changed the way classes were organised. The students worked with the EULALIA materials in groups of several people. In addition, they were encouraged to work with the materials independently at home, which was appreciated by the participants, some of whom valued the possibility of quiet work with the audio recordings. I2

The co-creation approach made our students in a way feel responsible for the content taught, they discussed the content with each other, they interacted differently, and they looked for the best solutions. They simply had fun. I3

In terms of the benefits of working with the EULALIA app, the interviewees highlighted the engagement of different senses: kinetic, visual and audio.

Working with EULALIA means, for me, organising varied stimuli for my students and myself. The tasks engage multi-sensory intelligences. I3

The participating teachers appreciated the topics that the EULALIA application covered and that it helped them to organise traditional classroom activities in a new, attractive way.

I liked the way the students felt engaged and motivated throughout the project, which gave them many opportunities to develop in an educationally attractive way. I1

EULALIA materials contain a solid portion of knowledge that cannot be found in commonly used textbooks. They teach what is regional, local and unique. Apart from the obvious benefits, such as learning vocabulary or listening skills, the materials help them to find their place, and to understand the local culture and history. I2

Regarding students’ engagement, the teachers working with the EULALIA application noticed an increased commitment to active participation in classes, which was linked to their tasks in the project (co-creation):

The students are more motivated with the co-creation experience than the use of the app alone. For the students who participated in the co-creation activities, it was very interesting. I2

The results and ease of coping with communicative situations were much better in the EULALIA group than in the reference group. I1

The teachers also underlined the improvement in results not only in terms of knowledge, but also in terms of the development of social skills, and cooperation in the classroom.

At the end of the project it turned out that the students in the EULALIA group had a richer vocabulary and better pronunciation. They are better at listening comprehension and are quite good at dealing with issues related to local culture and history (whereas the other students did not develop such skills). Moreover, students from the EULALIA group were better integrated, more willing to work in groups and enthusiastic about educational experiments. This, in turn, had an impact on the atmosphere in the classroom and therefore on the results. I1

Also, the students themselves, when asked by the lecturers about their reflections after the experience of working with the EULALIA application, admitted that they felt more involved when using the engaging digital application. Their active involvement increased not only because the application was an innovation in itself, but also because it was a part of the collaborative project which they had participated in. When asked why this was the case, the participants responded as follows:

The application was close to me because I felt involved in the project from the beginning. I5

I liked very much that the application dealt with those aspects that were the most challenging for me in a foreign country. Using the app made me feel very motivated! I9

The activities with the application were varied and we could often work together. I12

My engagement in the project was reflected in my attitude to the class with EULALIA. I5.

Summary

In summarising the outcomes of the quantitative study and supplementing them with the subjective feedback from lecturers regarding their experience with the application, it can be affirmed that the EULALIA application, designed for language learning and cultural immersion in a foreign country, proves to be a concept meriting ongoing development. Notably, participants using the application exhibited a statistically significant increase in knowledge, surpassing that of the reference group. The application additionally exerted a positive influence on the classroom atmosphere, contributing to heightened active involvement of the learners. These findings collectively underscore the viability and effectiveness of the EULALIA application, warranting further exploration and refinement of its instructional potential.

Discussion

The study aimed to assess the impact of the use of augmented reality on the effectiveness of teaching cultural elements from two perspectives: effect on perceived cultural knowledge of a host city (RQ1), and the improvement of workflow during classes from the lecturers’ point of view (RQ2). The main conclusion from the results brings to a positive correlation between the use of TUIs, and mobile learning, and increased cultural competence. In the long-term perspective, these observations may serve as a basis for a more extensive analysis of the relationship between the use of tangible interfaces and cultural learning in an international setting.

Regarding RQ1, conducted research, relying on self-assessment, indicated that the use of EULALIA app and the gamified approach may have a direct effect on learning compared to traditional lessons. The results have shown a statistical improvement in the group that used EULALIA. This improvement is also observed for the reference group, that attended traditional learning path, but with a minor confidence (95%). The EULALIA approach seems to promote a relevant improvement of the cultural knowledge of the city where a student is hosted. Also the common teaching methods reflects an impact, but using the innovative approach of EULALIA, the effect is stable and not limited to a single case. The data demonstrates a significance in the improvement in the results both for the test group (p < 0.001) and for the reference group (p < 0.005), but with a greater improvement in the case of the students that use the EULALIA application. The results are statistically significant, nevertheless the impact could be limited by technical issues related to the prototype phase of the application (i.e., bugs, software crashes, etc.)—several such concerns were voiced by the teachers and learners using the application. Therefore, more research on this issue is certainly needed. In particular, in this study the authors measured the impact of the whole EULALIA approach, without a separate analysis of the effects of the co-creation component and the presence of augmented reality features. Such analysis will be undertaken in future studies.

The observed need for a digital and interactive approach in the classroom is in line with the research conducted by Henrie et al. (2015). Students are exposed to numerous incentives connected with widespread digitisation in everyday life, and this also refers to the classroom reality, where the simulation of authentic situations using digital devices encourages and motivates students, as stressed by the teachers interviewed. The elements mentioned by Byram (1997) can be integrated into the application not only to make the learning more enjoyable, but also to make it more effective. In this context, the use of Tangible User Interfaces is promising, being in accordance with the meta-analysis by Li et al. (2022) that shows how tangibles scaffold learning, change learning behaviour, and improve learning emotion. This goes beyond previous studies that showed the usability potential of the EULALIA usage (Chinzer et al., 2023).

The results of the surveys and interviews presented here also confirm the initial conclusions of the GALLUP study (2018), mentioning the lack of engaging instruments and methodologies at higher levels of education, which leads to a decrease in the active involvement of older students. Students faced with an interesting and stimulating way of acquiring knowledge were more eager to work deeper on the issue, and in the end received better results. In the study, the author focused on the teacher’s perspective but for the better understanding of motivation incentives the further research considering also the student’s perspective could be beneficial.

This provides food for thought in the context of activities and practicality of tasks designed for older groups of students. Both the students and the teachers expressed interest in using interactive applications. More specifically, the interviews with the lecturers indicate that the EULALIA approach is highly accepted and could represent a relevant pedagogical innovation in second language courses. Interest is related to class engagement, which is triggered by the gamification process and the continued involvement of the students, who are active and follow the learning-by-doing approach (Roberts, 2012). Nevertheless, it is important to better understand the dropout in the test group for the students with lower language proficiency (A1). Dedicated research will further explore the issues which emerged and the reasons for this loss.

An interesting observation from the research is the fact that the students from the EULALIA group after the testing period declared a strong preference for learning with the Collaborative and Integrative approach, which was not the case in the reference group.

Conclusions

Due to the difficulties caused by COVID-19, the research needs to be extended in the future (both in terms of the number of participants and the length of the testing period) but the results collected so far indicate positive indicators of the use of augmented reality and the co-creation approach in language learning in universities regarding the improvement of cultural knowledge about the host city. The data shows a positive effect in the usage of the EULALIA application. As the scenarios were partly created by students for students, and covered a range of material not typically included in textbooks of high interest to the audience, they influenced the level of interest of that audience in the material presented. Co-creation of the contents on the augmented reality tools appears a promising strategy to involve students in active learning, also for those students that have limited knowledge of the tools. As also emerged in the interviews, the strength is not connected to technological innovation but in the pedagogical approach that includes innovative tools. In this way, the inclusion of students in the co-creation is broad, starting from a pencil-and-paper task that involves the development of new content. Student curiosity and the development of critical awareness have a crucial role in the educational success of both teachers and students. The conducted study provides both practical and methodological implications. The applied methodology can itself be a starting point for further long-term studies.

The results achieved can be successfully applied as a component of a simulation in specialised foreign language classes, where students can be confronted with authentic situations in which they will be making their own decisions. Additionally, the multisensory nature and open formula (individual creation of new scenarios) are elements which make it possible to use what is created in class as a form of testing the acquired information in practice, also in relation to classes other than foreign languages. Finally, augmented reality helps to foster the need for self-regulated learning in learners and taking responsibility for their own learning process.

An interesting impact regards the application of tangibles in higher education for language learners and cultural heritage, that represents an innovative approach. Tangibles are often applied to first grades of school or for the support of disabled people. This paper shows some initial results that do not cover only the usability of the platform, for higher educational purposes.

Further research on the topic will include an analysis of how activities reinforcing student engagement can be permanently implemented into a curriculum, and not only as part of a project. In a future long-term study, it would be particularly interesting to consider different age groups in order to observe possible individual differences and the impact of the co-creation principle on the results of adolescents and adults (not students). In addition, a long-term study needs to consider the digital competence of the participants, and the evolution of use of the application during a semester, to prevent any effect of extemporaneous intervention, namely the wow-effect. Another element worth investigating would be a comparison between face-to-face and completely remote scenario co-creation using available platforms. In this way, the study could be carried out in the future, irrespective of possible situations that prevented meeting. The pandemic made it very difficult to achieve the original aims of the described study, but on the other hand it was also an interesting experience in the context of building one’s own creativity and working in a virtual environment, so it is worthwhile to include a virtual creation environment in the context of future research with scenario co-creation.

Limitations

The long-lasting COVID-19 pandemic had an enormous impact on the study. First of all, many students were absent due to sickness, so the teaching process was not as effective as in the traditional face-to-face classroom. Initially, the number of students participating in the survey was restricted, as some of the students decided to return to their home country in the middle of the academic year. Also, the majority of classes during the research period were conducted online or in a hybrid form, due to the partial closure of the universities. The study was planned with students coming to the project partners’ countries as part of Erasmus mobility, unfortunately, in the face of the pandemic, this movement was almost completely halted, so the research sample collected was the largest possible. It is worth mentioning that part of the study (the pre-test) took place when very strict restrictions were not yet in place, then some participants decided to return to their home country, and some could not participate in the study due to illness. All this contributed to the fact that the results of the post-test were smaller, and due to anonymization it was not possible to exclude those who did not participate in the second part of the survey due to illness.

Moreover, while interpreting the findings of the study, it is essential to consider that both the test and reference groups were randomized. However, their initial knowledge of the cultural aspects of the host city was not equal. The statistically significant increase in knowledge observed in the test group may, in part, be attributed to a lower initial level of understanding. It is crucial to acknowledge these disparities and recognise that the study could benefit from replication in a more stable environment, devoid of a pandemic crisis, to establish a more informed and balanced distribution between the test and reference groups. Such a recalibration would contribute to a more nuanced evaluation of the intervention’s impact on participants’ cultural knowledge.

It would be worthwhile to carry out a more long-term study using augmented reality and to test its impact on students’ motivation: the pilot study helped to show some relevant points that should be extended in future studies (i.e., considering the effect of AR and the effect of co-creation independently). Another point to be examined in the future study lies in the involvement of lecturers. In this study they were free to support the students in the creation of the augmented maps. In future research, their role should be highly controlled to limit the influence on the results. As in this study, the goal is to apply the AR tools in more countries, disseminate the results and involve more higher education institutions, and make the collected data available.

Appendices

Appendix 1

PRE-Survey

The questionnaire is part of the Erasmus + funded project, EULALIA. The project aims to improve/integrate the learning methodologies of the university language centres for Erasmus students through the development of innovative and inclusive learning tools based on the paradigm of Mobile Learning and Game-Based Learning methodology and the application of Tangible User Interfaces (TUIs). We count on your support. By filling in the survey, you agree to Google’s privacy statement. Any personal data collected in this survey may be transferred to various countries, including the United States and other locations Google has offices in

[Question 23 was asked only in the EULALIA-Group].

1.:

Which year were you born?

2.:

What is your first/native language?

3.:

Are you an Erasmus student?

• Yes    No    Other

4.:

Which institution do you currently study at?

5.:

Which language do you study?

6.:

What is your current level of understanding the language you study?

• A1    A2    B1    B2    C1    C2

7.:

On a scale, where would you place yourself to describe your current knowledge about the culture in your host city?

• 0–10

8.:

I expect to be taught the following cultural aspects through my language course [Local cuisine]

• Strongly disagree disagree neutral agree strongly agree

9.:

I expect to be taught the following cultural aspects through my language course [Geography]

• Strongly disagree disagree neutral agree strongly agree

10.:

I expect to be taught the following cultural aspects through my language course [History]

• Strongly disagree disagree neutral agree strongly agree

11.:

I expect to be taught the following cultural aspects through my language course [Literature]

• Strongly disagree disagree neutral agree strongly agree

12.:

I expect to be taught the following cultural aspects through my language course [Sport]

• Strongly disagree disagree neutral agree strongly agree

13.:

I expect to be taught the following cultural aspects through my language course [Values]

• Strongly disagree disagree neutral agree strongly agree

14.:

I expect to be taught the following cultural aspects through my language course [Differences in pragmatics (i.e., politeness)]

• Strongly disagree disagree neutral agree strongly agree

15.:

At the end of the course, I will know [about the university history]

• Strongly disagree disagree neutral agree strongly agree

16.:

At the end of the course, I will know [how to find my way around campus]

• Strongly disagree disagree neutral agree strongly agree

17.:

At the end of the course, I will know [how to navigate myself in the city]

• Strongly disagree disagree neutral agree strongly agree

18.:

At the end of the course, I will know [the name of at least one famous person from the area]

• Strongly disagree disagree neutral agree strongly agree

19.:

Please rank how you prefer to learn

• (a) [Constructivism or Constructivist]

  • preferred approach neutral rather not

• (b) [Collaborative]

  • preferred approach neutral rather not

• (c) [Enquiry-Based]

  • preferred approach neutral rather not

• (d) [Integrative]

  • preferred approach neutral rather not

20.:

Why did you rank the way you did?

21.:

Digital tools should be introduced to improve teaching and learning

• strongly disagree disagree neutral agree strongly agree

22.:

On a scale, where would you place yourself to describe your digital skills?

• 0–10

23.:

Do you expect that your digital skills will evolve through your language learning course?

• Yes No

Appendix 2

POST-Survey

The questionnaire is part of the Erasmus + funded project, EULALIA. The project aims to improve/integrate the learning methodologies of the university language centres for Erasmus students through the development of innovative and inclusive learning tools based on the paradigm of Mobile Learning and Game-Based Learning methodology and the application of Tangible User Interfaces (TUIs). We count on your support. By filling in the survey, you agree to Google’s privacy statement. Any personal data collected in this survey may be transferred to various countries, including the United States and other locations Google has offices in.

[Questions 24–33 were asked only in the EULALIA-Group].

1.:

Which year were you born?

2.:

What is your first/native language?

3.:

Are/were you an Erasmus student during the last current semester??

• Yes    No    Other

4.:

Which institution do/did you study at?

5.:

Which language do/did you study?

6.:

What is your current level of understanding of the language you studied after taking the course?

• A1    A2    B1    B2    C1    C2

7.:

On a scale, where would you place yourself to describe your knowledge about the culture in your host city?

• 0–10

8.:

I have been taught the following cultural aspects through my language course [Local cuisine]

• Strongly disagree disagree neutral agree strongly agree

9.:

I have been taught the following cultural aspects through my language course [Geography]

• Strongly disagree disagree neutral agree strongly agree

10.:

I have been taught the following cultural aspects through my language course [History]

• Strongly disagree disagree neutral agree strongly agree

11.:

I have been taught the following cultural aspects through my language course [Literature]

• Strongly disagree disagree neutral agree strongly agree

12.:

I have been taught the following cultural aspects through my language course [Sport]

• Strongly disagree disagree neutral agree strongly agree

13.:

I have been taught the following cultural aspects through my language course [Values]

• Strongly disagree disagree neutral agree strongly agree

14.:

I have been taught the following cultural aspects through my language course [Differences in pragmatics (i.e., politeness)]

• Strongly disagree disagree neutral agree strongly agree

15.:

Because of my language course, I know [about the university history]

• Strongly disagree disagree neutral agree strongly agree

16.:

Because of my language course, I know [how to find my way around campus]

• Strongly disagree disagree neutral agree strongly agree

17.:

Because of my language course, I know [how to navigate myself in the city]

• Strongly disagree disagree neutral agree strongly agree

18.:

Because of my language course, I know [the name of at least one famous person from the area]

• Strongly disagree disagree neutral agree strongly agree

19.:

Please rank how you prefer to learn

• (a) [Constructivism or Constructivist]

  • preferred approach neutral rather not

• (b) [Collaborative]

  • preferred approach neutral rather not

• (c) [Enquiry-Based]

  • preferred approach neutral rather not

• (d) [Integrative]

  • preferred approach neutral rather not

20.:

Why did you rank the way you did?

21.:

Digital tools should be introduced to improve teaching and learning

• strongly disagree disagree neutral agree strongly agree

22.:

On a scale, where would you place yourself to describe your digital skills?

• 0–10

23.:

Did your digital skills evolve through your language learning course?

• Yes   No

24.:

I think that I would like to use this APP again

• Strongly disagree disagree neither agree nor disagree agree strongly agree

25.:

I found this APP unnecessarily complex

• Strongly disagree disagree neither agree nor disagree agree strongly agree

26.:

I thought this APP was easy to use

• Strongly disagree disagree neither agree nor disagree agree strongly agree

27.:

I think that I would need assistance to be able to use this APP

• Strongly disagree disagree neither agree nor disagree agree strongly agree

28.:

I found the various functions in this APP were well integrated

• Strongly disagree disagree neither agree nor disagree agree strongly agree

29.:

I thought there was too much inconsistency in this APP

• Strongly disagree disagree neither agree nor disagree agree strongly agree

30.:

I would imagine that most people would learn to use this APP very quickly

• Strongly disagree disagree neither agree nor disagree agree strongly agree

31.:

I found this APP very cumbersome/awkward to use

• Strongly disagree disagree neither agree nor disagree agree strongly agree

32.:

I felt very confident using this APP

• Strongly disagree disagree neither agree nor disagree agree strongly agree

33.:

I needed to learn a lot of things before I could get going with this APP