Development and validation of a secondary vocational school students’ digital learning competence scale

Abstract

The rapid advancement of digital technology has not only affected the world of work but also students' learning. Digital learning competence (DLC) is one of the essential skills students need for effective learning in a digital environment. Despite the significant presence of secondary vocational school students in China, constituting one-third of the high school demographic, research on their digital learning needs remains sparse. Addressing this gap, this paper attempted to propose the elements and structural model of digital learning competence for secondary vocational school students (V-DLC). A corresponding questionnaire was compiled, and an analysis was carried out with 872 valid survey data of secondary vocational school students achieved by convenient sampling. A five-factor model for the V-DLC was established through exploratory and confirmatory factor analyses, cross-validity, and criterion validity tests. This paper suggests that evaluating students' digital learning competence in secondary vocational schools can be achieved by considering the dimensions of cognitive processing and reading, technology use, thinking skills, activity management, and will management, combined with students' learning experiences in school and other fields. Given the global focus on digital learning competence, this framework will pave the way for empirical research on digital learning and guide the enhancement of student learning ability in vocational settings, adapting to the digital era. Furthermore, transitioning to a digitalized vocational education system is essential for preparing students for a digitally-driven workforce, aligning with modern job market demands and global trends.

Introduction

Digitalization and digital technologies are increasingly crucial in vocational education (Cattaneo & Barabasch, 2017; Cattaneo et al., 2022a, 2022b). Digital transformation reshapes working environments swiftly and continuously (Belaya, 2018; Dobricki et al., 2020; Enochsson et al., 2019), the integration of digital technologies into professional practices as tools for work alters demand on workers’ skill requirements and their necessitates proficiency with diverse technologies (Euler & Wilbers, 2019; Säljö, 2021; Castells, 1996; Karanasios & Allen, 2014; Willermark & Islind, 2023). Vocational education should aim to cultivate students’ vocational competencies, preparing them for evolving methodologies, tools, and procedural shifts in professional settings (Persson, 2020) and entrance into a dynamic world (Collins & Halverson, 2018). On the other hand, digital technologies are expanding learning opportunities beyond the classroom, blending formal and informal learning (Zhuang et al., 2017), and serving as learning tools to shape new pedagogical approaches in schools and influence students' daily learning experience(Euler & Wilbers, 2019; Collins & Halverson, 2018).

As digital natives, vocational school students' behaviors are deeply influenced by the progress of new ICT and digital technologies (Alexandru & Scoda, 2020), and information, media and technology skills, life and career skills, and learning and innovation skills have become the essential 21st-century skills for students (Partnership for 21st Century Skills, 2009). In China, secondary vocational schools, which serve one-third of high school students (Ministry of Education of China, 2023), are at the forefront of this digital transformation. The national strategy for digitalization (Ministry of Education of China, 2017) is accelerating the integration of digital technology into vocational education, with a focus on enhancing students' digital vocational competencies, digital learning competences, and comprehensive information literacy across disciplines. This integration has transformed vocational school students’ learning scenes, tools, and methods, incorporating new digital technologies such as VR and AR (Beskrovnaya et al., 2020), and the widespread adoption of virtual teaching space (Görl-Rottstädt et al., 2022). However, to effectively utilize digital technology tools for learning, to achieve the goal of learning more, faster, deeper, and at a lower cost (Vander Ark, 2011), as well as to be ready for professional life and citizenship (Carlsson & Willermark, 2023), students must possess digital learning competence.

Vocational school students' digital learning competence can help them adapt to digital learning and improve the learning effect. Learning is a complex, interactive, and dynamic process (Deakin Crick et al., 2015; National Academies of Sciences, Engineering, and Medicine, 2018) and a journey of inquiry from purpose to performance (Deakin Crick, 2014). Digital learning can be viewed from different perspectives, such as a learning approach (White, 1983), a learning process (Waller & Wilson, 2001 in Kumar, 2014), or a learning experience (Horton, 2011). But in a general sense, digital learning can be understood as the way learners engage in the learning process within a digital learning environment, by utilizing digital tools and educational resources through a digital learning approach (Li, 2001) to foster learning’s effectiveness and efficiency (Rosenberg & Foshay, 2002; Weinstein et al., 2011). Digital Learning Competence (DLC) can be characterized as a combination of knowledge, skills, and attitudes that empower students to achieve efficient and effective learning through digital tools within digital learning environments (Yang et al., 2021; Pedaste et al., 2023).

Currently, research on digital learning competence frameworks scales and assessments primarily focuses on K-12 education and higher education, with relatively few studies specifically targeting secondary vocational school students, particularly in the context of digital transformation. Many previous researches have been carried on the digital competence of students of primary, secondary, and higher education (such as Aesaert et al., 2015; Perifanou & Economides, 2019a, 2019b; Tzafilkou et al., 2022; Zhao et al., 2021;), for teachers and educators (such as Touron et al., 2018; Revuelta-Domínguez et al., 2022), and for organizational learning (such as Giannakos et al., 2022). Several studies have focused on the digital competence of vocational education teachers (such as Batz et al., 2021; Marín & Castañeda, 2022; Cattaneo et al., 2022b) and students (such as Wild & Schulze Heuling, 2020), especially for vocational secondary schools students (such as Patmanthara & Hidayat, 2018; Warno, 2020; Haryani, 2023; Delima et al., 2022). Some research also concerns the digital learning competencies of students (such as He et al., 2020; Tarigan et al., 2023; Kallas & Pedaste, 2022), teachers, and educators (Department of Basic Education, Republic of South Africa, 2019), but few had been found about vocational secondary school students. Especially in China, most of the research related to students' digital learning competence has focused mainly on K12 education, followed by higher education, with a lack of research in other educational areas such as vocational education, teacher education, and adult education (such as He & Zhu, 2017; He & Zhu, 2017; Li, 2014; Li & Zhang, 2022; Yang et al., 2021; Zhou & Zhang, 2014). However, few empirical studies have been undertaken on students in secondary vocational schools. In particular, of those studies that focus on the importance of digital competence in vocational education, most focus on how educators or teachers can fluently and effectively use digital technology to ensure students' participation and development of digital competence (Cattaneo et al., 2022a, 2022b). Some scholars have primarily focused on the components of learning competence and enhancement strategies for students in secondary vocational schools. For instance, Wang (2024) and Zhou (2018) have identified the elements constituting the learning competence of these students from the perspective of integration between general and vocational education, and the context of the intelligent era, respectively, discussing strategies for improvement.

Based on the above research, this study attempts to construct a digital learning competency framework for secondary vocational school students(V-DLC) based on the unique characteristics of the digital generation learners and the three elements of learning competency. It also develops and validates the corresponding scales. To achieve this goal, this study attempts to answer the following two research questions:

• RQ1: What are the factors and structure of V-DLC?

• RQ2: Is the digital learning competence questionnaire for secondary vocational school students based on the V-DLC reliable and effective?

Structure and elements of V-DLC

Framework of DLC in literature

In previous studies, scholars have elucidated the framework of digital learning competence from diverse perspectives. Commonly, many frameworks propose an incorporation of basic to advanced skills. Specifically, they recognize DLC as inherently multi-dimensional, integrating cognitive, technical, and attitudinal elements. Most frameworks highlight the importance of instrumental knowledge and skills, advanced digital skills, and a positive attitude towards digital applications. However, the specifics of these components and their emphasis can vary widely, which can differ based on the focus of the research, the targeted educational level, and the perceived needs of learners in a digital society. For example, some frameworks focus on basic skills like information processing, while others emphasize more complex competencies such as online communication and teamwork. Additionally, the approaches to assessing these competencies differ, with some researchers developing detailed practical assessment tools. The following paragraphs will expand on this.

Digital learning competence is intricately linked to the process of information processing. According to different ways of using information technology, Zhong and Yang (2001) classified digital learning competence into the competence of acquiring information, using ICT tools, generating information, processing information, collaborating learning with information, creating information, utilizing the benefits of information, information ethics, and immunizing information. In contrast, Peng (2002) and Xu and Lu (2014) reduced information ethics and the ability to generate information, added the ability to express information, and categorized digital learning competence into eight elements. Wang (2017) and Luo's (2018) division is more streamlined, subtracting collaborating with information and utilizing the benefits of information, which constitute the six elements of digital learning competence.

Digital learning competence is not a mere information processing process, but a comprehensive ability related to multiple dimensions such as cognition, skills, and attitudes. Wang et al. (2013) believed that the elements of digital learning competence include instrumental knowledge and skills, advanced knowledge and skills, and attitude toward application. Lu et al. (2018) believe that the digital learning approach includes the adaptive ability with digital features, the ability to process tacit knowledge, and the meta-learning ability. Dondi et al. (2006) proposed that in digital learning, learners should have corresponding experiences that support learning, strong motivation to learn, online communication skills, the ability to manage complex situations, open-mindedness, teamwork, as well as organizational and monitoring skills. He et al. (2020) mentioned that digital informal learning behaviours could include three formative dimensions: cognitive, meta-cognitive, and social and motivational learning. Researchers also describe the components of DLC from the perspectives of digital learning environment, resources, and learning methods.

Researchers have established a framework of digital learning competence from different levels and dimensions. For example, Yang (2019) regarded digital learning awareness, behaviour, management, and evaluation as the four significant aspects of college students' digital learning competence. Yang (2018) divided digital learning competence into three levels of foundational, generative, and sustaining elements based on students' learning characteristics, and specified its elements into technology application force, digital learning attitude, information processing force, learning decision-making force, independent learning force, and learning innovation force. Li (2014), on the other hand, analyzed the digital learning competence of college students and divided it into five levels: learning awareness, learning technology, learning behavior, learning management, and learning evaluation. Martin et al. (2020) identified four critical constructs essential for effective e-learning: attributes of online students, encompassing self-regulated learning and academic self-efficacy; proficiency in time management, highlighting the importance of effective time utilization; technical skills, including computer, internet, and information-seeking abilities; and communication skills, emphasizing the readiness to engage in communication during e-learning.

In discussions surrounding digital competence, the Digital Competence Framework for Citizens (DigComp) is often regarded as a foundational guide for establishing a shared understanding of digital competence (European Commission, Joint Research Centre, 2022). Previous studies have assessed digital learning competence based on DigComp and incorporated new components like attitude dimensions. For example, a study by Kallas & Pedaste (2022) utilized an assessment tool they developed and validated. This instrument is designed to evaluate the digital competence of primary and lower secondary school students across ten dimensions. These include social aspects, perceived control, behavioral attitudes, behavioral intention, creation of digital content, programming digital content, digital communication, operation of digital tools, digital safety for self and others, and legal behavior in the digital realm. These dimensions can be categorized into three distinct subcategories: firstly, attitudes towards using digital devices, knowledge and skills, and behaviors within the digital world. Another K-12 student standard in digital learning, set by ISTE (2016), has also been used as the K-12 students' digital learning standards or competencies.

Scholars also established a competency hierarchy framework according to the level of digital learning competence. Regarding the elements of DLC, Martin (2006) and others have identified three levels of DLC: basic, applied, and innovative. For example, according to the level of competence, Li (2016) classified digital learning competence into four levels: beginner, intermediate, higher, and advanced, and specifies the ability level of each level in detail. Among them, learners at a higher ability stage can re-screen and integrate the fragmented information presented by various media to form new content, and comprehensively use multiple digital tools to carry out research-based learning and form conclusions. Learners in the advanced ability stage can develop digital tools to improve learning and research efficiency and apply critical thinking to identify problems in cognitive, communication, collaborative learning, research, and other learning activities.

Specific characteristics of DLC in vocational schools

Supported by the latest advancements in information and intelligent technology, DLC for secondary vocational school students exhibits unique characteristics and elements distinct from those of college students. Specifically, this differentiation is manifested through the utilization of diverse digital resources and equipment that focus on the acquisition, transfer, and improvement of skills of workers at different stages (Gosling, 2021), connecting the education system (including integrated vocational education, general education, and continuing education) and the world of work (Helms Jørgensen, 2004).

Emerging technologies, such as the Internet, big data, cloud computing, and so on, are being fully integrated into all walks of human life, including the economy, politics, and culture, with new concepts, new forms, and new modes, which have had a substantial impact on the form of social occupations and have given rise to new forms and modes of employment. It poses challenges to the field of vocational education and also brings opportunities.

According to the "Report on the development of vocational education in China (2012–2022)" published by the Ministry of Education of China (2022), in the past ten years, more than 90% of vocational schools have built campus networks that run smoothly and have complete functions; more than 85% of vocational schools have built digital campuses according to standards. Several online course platforms have been built, 203 national-level professional teaching resource libraries for vocational education have been built, and 992 high-quality video open courses and 2886 national-level high-quality resource sharing courses covering 12 disciplines such as arts, sciences, industry, agriculture, and medicine have been developed. Information-based teaching in vocational colleges is becoming the new normal. What’s more, the survey of "Vocational Education Informatization Development Report of China 2021" (Institute of Education of Tsinghua University, 2022) shows that more than 74% of vocational school teachers are using information technology to teach, students’ recognition of the learning effect of information technology exceeds 58%, and their recognition of virtual simulation training is as high as 90%. Digital technology is also widely used in teaching, learning, management, evaluation, research, and other scenarios in vocational colleges to build a smart campus for vocational education.

Conceptualised V-DLC dimensions

Foundations of V-DLC

The foundation for V-DLC conceptualization is grounded in multiple sources. The first is a comprehensive review relating to V-DCL. The second is the three aspects of goal, will, and ability of learning competence. The third is the Biggs’s “3P” model. The fourth is the Framework for 21st Century Learning, and the last is the Model of Strategic Learning.

From a functional perspective, digital learning competence can be perceived as a specialized form of competence framed within the context of demands and tasks, integrating specialized cognitive aspects with motivational and behavioral elements (Rychen & Salganik, 2000; Winkelen & McKenzie, 2011a, 2011b). Based on the learning process and the ability, motivation and opportunity framework (Kellner et al., 2019), this approach underscores the multifaceted nature of competencies, integrating cognitive, emotional, and managerial aspects. The learning competence of secondary vocational school students can be seen as the intersection of goal(motivation), will(perseverance), and ability (including knowledge and practice) (Zhang, 2004). Ability is the basis for students' effective learning, including the knowledge and practical basis needed for learning (Gauthier, 2013). To some extent, goal setting can determine students' learning results, and cognitive and behavioral responses to achieve goals can stimulate students' learning experiences (Locke & Latham, 2002). Will and motivation are inseparable, and self-efficacy enhances an individual's commitment to a goal (Stajkovic & Luthans, 1998). A higher level of learning motivation and strong will can stimulate, maintain, and promote learning activities (Filgona et al., 2020).

Biggs's et al. (2001) "3P" model of the "Presage Process Product" can help to consider students’ learning from students' learning ability, process, and results. "Presage" mainly refers to students' characteristics, including existing knowledge and skills and personal preference for learning; "Process" refers to the learning process of students, including the choice of learning motivation and strategies, mainly the methods adopted by students to complete certain tasks; "Product" is the learning gained by students.

These two frameworks can explain each other to a certain extent. The ability includes essential knowledge and skills, the basis for students to carry out learning, and the front part of learning. Goals stimulate students' behaviors; students adopt various methods and strategies to manage their learning behaviors and complete tasks. Will, motivation, and perseverance are closely related, which can stimulate and maintain deep learning.

Besides, the skills required for education and the modern workforce are commonly referred to as 21st-century skills, reflecting the evolving demands of today's economy. The approach to defining digital skills has evolved from a technical focus to a broader perspective, including content-related or higher-order skills (Claro et al., 2012). The significance of these skills in meeting the demands of 21st-century workers has been firmly established (Van Laar et al., 2020). The analysis reveals a widespread consensus on the importance of competencies in communication, collaboration, ICT-related skills, and social or cultural awareness. Additionally, most frameworks identify creativity, critical thinking, problem-solving, and the ability to produce relevant and high-quality outputs as key competencies in the twenty-first century. The primary distinctions among these frameworks arise from their categorization and grouping of competencies, the significance assigned to each, and the emphasis on ICT skills as central to these abilities, as noted by Voogt and Roblin (2012). For example, the Partnership for 21st Century Skills (P21, 2009), a collaborative entity between government and corporate sectors, categorizes skills into three distinct types: learning and innovation skills (learning skills); information, media & technology skills; and life & career skills (literacy skills, and life skills). The international research project Assessment and Teaching of 21st Century Skills (ATC21S) resulted in 10 skills grouped into four categories: ways of thinking (creativity and innovation; critical thinking, problem-solving, and decision-making; learning to learn and metacognition), ways of working (communication; collaboration-teamwork), tools for working (information literacy; ICT literacy) and living in the world (citizenship; life and career; personal and social responsibility) (Binkley et al., 2012). In this paper, the structure of these 21st-century skills-encompassing ways of thinking, working, and working tools, particularly in navigating our globalized world—is adopted as one of the substantial foundations for the construction of V-DLC dimensions.

In the context of the rapidly evolving global economy and workforce, the demand for individuals to become strategic, self-regulated lifelong learners is increasingly recognized. This paradigm shift is foundational to understanding the Model of Strategic Learning (MSL), a framework critical to developing the Vocational School Students’ Digital Learning Competence (V-DLC). MSL delineates the essential learning elements into four key components: skill, will, self-regulation, and the academic environment. These components collectively form the theoretical basis for V-DLC. Weinstein and Acee (2018) emphasize the importance of this model by exploring approaches to teaching and assessing strategic learning. Their insights are instrumental in shaping strategies for fostering digital learning competences in vocational education settings.

Dimensions of V-DLC

As a complement to a comprehensive review of DCL conducted by our research team (Yang et al., 2021), which released that DLC consists of six dimensions, namely technology use, cognitive processing, digital reading skill, time-management, peer management, and will management, a comprehensive review of innovation, thinking skills, and other aspects of V-DLC has been undertaken in this study. Based on the above foundations, especially the ability, will, and goal of learning competence, combined with the learning preferences and characteristics of secondary vocational school students in the digital era, this study attempts to propose nine elements for the evaluation of secondary vocational school students' digital learning competence, as shown in the following Table 1.

Table 1 Formative framework of digital learning competence of secondary vocational students

In this paper, ability can refer to ‘the physiological and cognitive capabilities that enable an individual to perform a task effectively’ (Blumberg & Pringle, 1982). The twenty-first century skills are essential for working and learning, the ICT skills are the core of these skills (Voogt & Roblin, 2012), and cognitive abilities have been recognized as a crucial foundational element for acquiring a wide range of these skills (Van Laar et al., 2020). So, in this paper, the ability aspect of V-DLC mainly focuses on the categories of ways of thinking, tools of working and living in a world of 21st-century skills (Binkley et al., 2012). In other words, it mainly includes the technological and cognitive (including high-order cognitive skills) dimensions. The ability aspect of V-DLC can be described as three dimensions: technology use, cognitive processing, reading skills, and thinking quality.

Technology use is closely related to students' information literacy. It is about students' ability to use different technologies to learn, work, and innovate, including using technical tools to find, evaluate, store, retrieve, and apply different information (Catts & Lau, 2008). Technology use also reflects multimedia learning characteristics, including some stages: media selection, organization, integration, and creation. At the same time, students should be able to maintain their information security with technology. This study's “technology use” mainly includes media selection, organization, integration and creation, rich media cognition, and network security.

Cognitive processing deals with the information processing ability of secondary vocational school students in learning. Van Laar and colleagues (2020) propose that 21st-century skills and ICT skills are interconnected, potentially manifesting as ICT 21st-century skills or 21st-century digital skills. They also recognize cognitive abilities as a crucial foundation for acquiring many of these skills, highlighting the integral role of cognitive capacities in skill development. Cognitive processing broadly refers to the array of cognitive activities that generate and manipulate mental representations of information (Krch, 2011). The most basic cognitive learning strategies are rehearsal (e.g., repeating, copying, shadowing), elaboration (e.g., paraphrasing, summarizing), and organization (e.g., outlining, creating a hierarchy, Pintrich et al., 1993; Weinstein & Mayer, 1983). Combining the learning strategy scale of LASSI and MSLQ, and following the procedure of the information process), this study's "cognitive processing" mainly includes five aspects: selection points, rehearsal, elaboration, organization, and reflection.

Reading Skills are about students' reading ability. Reading refers to realizing personal goals, developing self-knowledge and potential, and participating in social activities by understanding, using, and reflecting on texts in both paper and digital resources. Reading encompasses a diverse array of cognitive competencies, ranging from fundamental decoding skills to understanding words, grammar, and broader linguistic and textual structures, as well as knowledge about the world, as outlined by the OECD (2013). Digital reading resources include online texts, short videos, and other forms. SQ3R-survey, question, read, recite, and review (Robinson, 1970), a well-known reading strategy in the world will help students to study better. In this study, "reading skills" include reading skills such as overview, question setting, active reading, retelling, and review. There are two main kinds of reading media: paper and digital resources.

Thinking is a purposeful activity unique to humans to understand and solve problems (Lin, 2007). Thinking skills mainly relate to students’ higher-order thinking skills (HOTS). As to HOTS, earlier research has defined them as encompassing analytical, synthetic, and evaluative abilities, skill development, estimation, generalization, creative thinking, decision-making, objective setting, and critical and systemic thinking, as noted by Dillon and Scott (2002), Miri et al. (2007), and Zohar and Dori (2003). Some studies have broken down HOTS into two main components: critical and creative thinking skills, as identified by Heong et al. (2011), Plan (2014), and Sulaiman et al. (2017); and logical thinking and reasoning skills, as per Marshall and Horton (2011). Other researchers like Apino and Retnawati (2017) and Lewis and Smith (1993) have categorized HOTS into four aspects: critical thinking, creative thinking, problem-solving, and decision-making thinking. Computational thinking (CT) and design thinking (DT) are essential methodologies for students to enhance their problem-solving abilities. Among them, Wing (2006) introduced the concept of Computational Thinking, defined as the ability to think at various levels of abstraction to solve problems within design systems, and more, further emphasized that CT is a vital skill for computer scientists and everyone. Initially emerging from design firms to aid in complex problem-solving, design and art colleges have broadly embraced design thinking as a key approach to nurturing students' creativity. DT represents an analytical and creative problem-solving process, focusing on empathy, divergent thinking, experimentation, prototype testing, and iterative design methods, as described by Razzouk and Shute (2012). This study's thinking skills mainly include creative, critical, computational, and design thinking, which correspond to the development of students' innovation, criticism, abstraction, and comprehensiveness (Li et al., 2020).

Will is related to individual motivation and profoundly impacts students' learning behavior and learning effect (Weinstein et al., 2011), which is embodied in this study as "will management." Motivation is often viewed as a driving force that guides, invigorates, and maintains behavior (Van Iddekinge et al., 2018), or as the readiness and eagerness of employees to undertake a task (Bos-Nehles et al., 2013). According to the social cognitive model of motivation, motivation includes expectation, value, and emotion. According to Pintrich et al. (1993), expectancy would consist of students' self-efficacy beliefs and their control beliefs for learning; value would encompass reasons for academic engagement, with subscales for intrinsic and extrinsic goal orientations, and task value beliefs; and affect would be operationalized through responses to a test anxiety scale that measures students' worries and concerns about exams. In this study, "will management" mainly consists of four parts: learning belief, motivation, self-efficacy, and test anxiety (Zhuang et al., 2018).

The goal is to imagine, plan, and promise the expected results of the problem. The process of achieving these goals is one of individual development, closely linked to the individual's capabilities in self-management, managing others, and task management (Huang & Zheng, 2005). Time management tendencies are often regarded as a personality trait (Huang & Zhang, 2001) and form a crucial aspect of self-management. In vocational education, learning goals are closely related to specific tasks, and the process of attaining goals is also the process of completing tasks and obtaining outputs, which is inseparable from individual management of the entire task. Starting from the task learning faced by vocational school students, this study focuses on four parts: time management, task clarity, partner management, and resource management.

Time management is about students' ability to use and arrange time. Completing tasks is manifested explicitly in activities such as actively setting goals, confirming priorities, allocating time, and checking results (Huang & Zhang, 2001).

Task management is about the students' clarity of specific tasks, mainly including mastering the task's particular requirements, analyzing the task's completion process, and clarifying the task's output. Task management encompasses using assistive tools and performance management (Huang & Zheng, 2005). This study assesses an individual's ability to analyze and plan learning tasks, including clarifying tasks, analyzing processes, and defining outcomes.

Partner management is students' ability to learn cooperatively with peers when achieving goals, including getting along with peers, conflict management, managing others, and accepting management (Huang & Zheng, 2005). In this study, partner management mainly includes cooperation with others, support and guidance to others, negotiation, and conflict management (Wang et al., 2009).

Resource management is about the students' ability to manage the available environment and resources, including managing time and study environment, effort management, peer learning, help-seeking, and so on (Pintrich et al., 1993). This study mainly reflects the setting of the learning environment, the use of learning tools, and social human resources.

Content validity of V-DLC dimensions

To further validate and revise the dimensions, this study collected feedback from five experts by using the self-compiled questionnaire through the Delphi method. The expert consultation questionnaire presents 20 indicators of the evaluation framework and collects the feedback data of experts for analysis. The five experts were teachers and administrators from different secondary vocational schools in Beijing. The experts were tasked with employing a 5-point Likert scale to assess the items' significance and offer their feedback regarding modifications and the inclusion of new items.

Data analysis focused on each item's mean, standard deviation, coefficient of variation, and full score rate. The screening of quality indicators was determined mainly based on the importance acknowledged by the experts and the coefficient of variation. For each item, it would be kept directly if the mean of importance acknowledged by the experts (the ratio of the number of experts scoring 4 and 5 in each item to the total number of experts) is greater than or equal to 75%, the average score is greater than or equal to 4, the standard deviation is less than 1, and the coefficient of variation is less than 0.2 (Zhang & Huang, 2010).

Eventually, all the indicators met the standards, of which 78% of the indicators had a mean score of 4.5, and 70% of the items had a full score rate of 75%. We obtained the secondary indicators of the digital learning ability assessment framework for secondary school students (see Table 2), which is consistent with the original assessment framework design.

Table 2 Secondary indicators of the Delphi method test

Research design

In this study, the development of the scale meticulously follows the structured methodology suggested by Boateng et al. (2018), encompassing the formulation of items, the construction of the scale, and its subsequent evaluation. This approach is akin to the research process where various elements of digital learning competence are identified and measurement items are meticulously compiled from diverse sources. For instance, the crafting of cognitive information processing items derives inspiration from the LASSI (Weinstein & Palmer, 2002) and MSLQ (Pintrich et al., 1991) scales. Similarly, the development of other components, such as reading skills, technology use, time management, task and partner management, creative and computational thinking, and critical thinking, integrates established frameworks and scales, reflecting a comprehensive and methodical approach to scale development.

Item development

While existing digital competence scales provide valuable foundations, there is room for enhancing their integration and relevance. This research builds upon these scales by broadening the scope of skills included and updating the content to reflect the rapid technological changes impacting today's educational environments. Moreover, the scale is specifically tailored to meet the unique needs of secondary vocational students, ensuring its applicability across diverse educational contexts.

This research is based on the composition and evaluation elements of digital learning competence and refers to the test questions from various sources to compile measurement items. Among them, cognitive information processing refers to the items of the LASSI (Weinstein & Palmer, 2002) and MSLQ (Pintrich et al., 1991) scales. Reading skills refer to the meaning of each step in SQ3R (SQ3R: Effective Reading, 2023) and are compiled in combination with the key points of PISA reading skills (OECD, 2019a, 2019b). The use of technology is combined with the European Digital Literacy Framework (Thompson, 2013), the Global Media and Information Literacy Assessment Framework (Ferrari, 2013), and the characteristics of digital natives. The time management items mainly refer to Huang and Zhang's (2001) Adolescence Time Management Disposition Inventory. Task management and partner management items are compiled according to their meanings and categories. Creative thinking items mainly refer to Williams’ (1980) Creativity Assessment Packet. Computational thinking items are compiled principally according to ISTE evaluation indicators and their categories (ISTE, 2019). And critical thinking items are compiled primarily by referring to the MSLQ scale (Pintrich et al., 1991), the design thinking is mainly combined with the DTDS scale (Tsai & Wang, 2021), as shown in Table 3.

Table 3 The items and their referred primary resources

Finally, a questionnaire with 80 items is formed, each using a Likert's five-point scale. In the questionnaire, students are also asked to make an overall evaluation of their learning performance in the form of self-description, including learning achievements, communication, and communication skills, innovative thinking skills, problem-solving skills, self-directed learning competence, and other indicators, and these indicators are taken as the indicators of the digital learning competence of secondary vocational students.

Data collection

In 2021, Beijing's 109 secondary vocational schools across 16 districts offered diverse programs, including general and adult junior colleges, vocational high schools, and vestibule schools. The secondary vocational schools’ enrollment ratio to general senior high schools was approximately 29:71. These schools were well-equipped, averaging 1.11 computers per student, and focused heavily on integrating digital tools to enhance educational outcomes. The reform efforts emphasized project-based and task-driven teaching methods, coupled with a robust use of online resources to support blended learning, thus aligning with Beijing's goal to foster holistic development and vocational proficiency in a digital era (Beijing Municipal Education Commission, 2022).

Between October and November 2021, the research team distributed questionnaires to survey the digital learning competence of vocational school students in Beijing. The survey employed a convenience sampling approach, selecting five vocational schools in Beijing, covering five districts: Changping, Haidian, Xicheng, Shijingshan, and Chaoyang, including three vocational high schools and two vestibule schools. Additionally, considering the context of the integration of the Beijing-Tianjin-Hebei region, a vocational school from Hebei province was also included in the research. The questionnaire was primarily distributed online, with 1210 students from six vocational schools participating, and 1210 valid student questionnaires were collected. From these, 872 questionnaires with complete responses to all 80 items, each with distinct answers, were selected for analysis.

Data analysis results

The collected data were analyzed using SPSS 21.0 and AMOS 22.0. Exploratory factor analysis (EFA) was utilized to ascertain the factor structure, while confirmatory factor analysis (CFA) was employed to establish the structural validity of the scale. The 872 responses were randomly divided into samples A and B of an equal number. Sample A was used for exploratory factor analysis, and sample B was used for confirmatory factor analysis.

Component analysis of V-DLC

Item reduction analysis

Before conducting exploratory factor analysis, item analysis is used to identify items that can be used for subsequent analysis. In this study, 436 data of sample A were used for item analysis of 80 items by extreme group test (T-test for independent samples of high and low groups) and homogeneity test (correlation coefficient between the modified item and the total score of the scale, scale α coefficient after deleting the item).

It was revealed that the differences between the mean values for each item measured by the high and low-scoring groups were all significant (p < 0.001), indicating good discriminative power for all 80 items. The analysis of the "Correlation between the modified items and the total scale score " showed that the correlation coefficients between each item and the sum of the other items ranged from 0.577 to 0.878, all exceeding 0.3. Additionally, there were five items, and after removing one, the Cronbach's alpha coefficient for internal consistency of the scale increased, indicating improvement. Therefore, these five items were deleted. The remaining 75 items demonstrated good homogeneity, suggesting that all 75 items can be used for exploratory factor analysis.

Exploratory factor analysis

Evaluating the suitability of the collected data is a prerequisite before conducting exploratory factor analysis. The Kaiser–Meyer–Olkin (KMO) test value of the scale was 0.978, and the Chi-square value of the Bartlett sphericity test was 32,417.703 (1770 degrees of freedom), reaching the significance level (p = 0.000). The scale data were suitable for exploratory factor analysis (Tabachnick & Fidell, 2007).

Without specifying the number of common factors, the principal component method, in conjunction with direct oblique rotation, was employed to extract five common factors with eigenvalues greater than 1. The cumulative variance contribution rate reached 75.824%. A check on the intercorrelation among items, factor loadings of items on each factor, etc., prompted a re-exploratory factor analysis. The number of items on each factor was balanced to simplify the model. Ultimately, 60 items were retained, extracting five common factors that accounted for a total variance of 75.824%. The internal consistency coefficient for each factor ranged from 0.861 to 0.973, showing good consistency, as shown in Table 4.

Table 4 Summary table of exploratory factor analysis results

The distribution of the five factors and 60 questions conforms to the assumptions on item structure during item design, as shown in Table 5.

Table 5 Five factors of V-DLC and their corresponding items

Dimension analysis of V-DLC

This study conducted first-order and second-order confirmatory factor analyses (CFA) on Sample B data to examine the five factors of V-DLC derived from exploratory factor analysis. Further investigation into the relationships between these factors was carried out to establish the structure of V-DLC. Cross-validity testing was performed using data from both Sample B and Sample A to validate that the model can be used to explain the overall survey data.

CFA for measurement model

In this study, first-order CFA was initially conducted on the measurement models of the five factors obtained from the exploratory factor. It was observed that the factor loadings within each factor model ranged from 0.548 to 0.903 except for one item, with all error variances being non-negative and reaching significance levels. Fit indices computation revealed that the absolute fit index RMSEA values for each factor's first-order model were below 0.08, and the goodness-of-fit indices GFI, AGFI, and CFI values were above 0.9. The first-order models for each of the five factors demonstrated a good fit, establishing the five-factor measurement model of V-DLC.

To further assess the structural validity of this five-factor measurement model, it is necessary to calculate convergent validity and discriminant validity. The composite reliability (CR) of the five factors was found to be in the range of 0.957 to 0.973, and the average variance extracted (AVE) ranged from 0.651 to 0.710. Additionally, factor loadings mainly were above 0.7. It indicated that all five factors possess convergent validity. Various methods can be employed to assess discriminant validity between dimensions. In this study, the approach of comparing the square root of AVE was used to determine discriminant validity (as shown in Table 6). The results confirmed that these five factors exhibited both convergent and discriminant validity.

Table 6 Convergence validity and differential validity of each factor

CFA for structural model

In the above analysis, some of the first-order factors exhibited high correlations. Therefore, this study further attempted second-order confirmatory factor analysis (CFA) to explore the relationships between these factors and establish an overall structural model. Based on the meanings of each factor and their interrelationships, several competing models could be established:

First-order single-factor model: A single factor simultaneously explained all observation items.

First-order five-factor uncorrelated model: The five factors were independent of each other.

First-order five-factor correlated model: The five factors were correlated with each other.

Second-order single-factor model: The five factors belonged to a single higher-order factor.

Second-order two-factors: The five factors belonged to two higher-order factors. Cognitive processing reading, and activity management belonged to one higher-order factor, while the other three belonged to another.

Second-order three factors: The five factors belonged to three higher-order factors. Cognitive processing and reading, and technology use belong to one higher-order factor. Will management was independent, while thinking skill and activity management belonged to one higher-order factor.

Second-order four-factors: The five factors belonged to four higher-order factors. Cognitive processing and reading constructed an independent factor, and technology use was a separate factor, will management was an independent factor, and activity management and thinking skills belonged to another higher-order factor.

By comparing the fit indices of these seven competitive structural models (as shown in Table 7), the differences in the indicators among the models were relatively small. Considering the original dimension division of the scale, the first-order five-factor correlated model demonstrates a relatively good fit.

Table 7 Fit indexes for second-order confirmatory factor analysis

The first-order five-factor correlated model, composed of cognitive information processing and reading, technological use, thinking skill, activity management, and will management, fits well with the actual data.

Cross-validity test of the model

To explore the stability of this model, Sample B was used as the calibration sample, and Sample A was used as the validation sample for cross-validity analysis. This aimed to test the measurement equivalence of the model. The study focused on utilizing the Incremental Fit Index (CFI) to assess differences between models, where |ΔCFI|≤ 0.01 indicated that the difference between the two models was not statistically significant (Cheung & Rensvold, 2002). Some studies also used ΔTLI ≤ 0.05 as a criterion for testing differences between nested models(Little, 1997). The results of the multi-group comparisons are presented in the Table 8. Since |ΔCFI|≤ 0.01 and ΔTLI ≤ 0.05 for all model comparisons, it could be concluded that the differences between the models were not significant. It aligned with the requirements of a moderate test proposed by Byrne(2016). The first-order five-factor model's measurement coefficients, structural coefficients, and measurement residual models exhibited cross-group validity, indicating that the two groups are equivalent. The model demonstrates stability, supporting further analysis.

Table 8 Summary table of multi-group invariance comparison

The results of those above first- and second-order CFA and the cross-validity testing indicated that the digital learning competence assessment framework for vocational school students can encompass five dimensions: cognitive information processing and reading, activity management, thinking skill, technological use, and will management.

Criterion validity test of V-DLC on learning performance

Learning is aimed at achieving corresponding learning performance. Based on the "21st Century Competency Framework for Adolescents"(Partnership 21st Century Skills, 2018), students' self-perception indicators, such as learning performance, communication, innovation, problem-solving, and self-directed learning competence, were used as the criterion. Using combined data from samples A and B, the correlation between various dimensions of V-DLC and these performance indicators was analyzed (as shown in Table 9). The results revealed a significant positive correlation (p < 0.01) between the dimensions of V-DLC and individual indicators of learning performance. It suggests that the V-DLC evaluation framework can somewhat reflect students' learning situations, indicating the model's utility.

Table 9 Correlation matrix between scores of various dimensions and criterion scores

Differences in subjects' V-DLC

Due to the convenience sampling method employed in this study, this study would focus on differences in V-DLC dimensions among participants from grade levels.

Overall, regarding the development of digital learning competence of secondary vocational students, the dimensions are ranked in descending order of mean scores: technology use, cognition processing and reading, thinking skills, activity management, and will management. It indicates that the participants are most proficient in applying information technology, while there is room for improvement in willingness management.

Through one-way analysis of variance (ANOVA), it was found that students of different grades show differences in the development of cognition processing and reading, technology use, thinking skills, activity management, and will management. Post-hoc comparisons revealed that second-grade students were significantly lower in terms of cognitive processing and reading skills than third and fourth-grade students (note: one senior technical school participating in the test has fourth-grade students). Regarding technology use, the scores of students in grade Two were significantly lower than those in the other three. Regarding thinking skills, the second-grade students had lower scores than the other three grades. In activity management and will management, the second-grade students were lower than the third-grade students.

Since this survey took place between October and November in the fall semester, it corresponded to the period when second-grade students had just completed their first year of vocational school, and third-grade students had just finished their second year. It could be observed that the development of second-grade students in various aspects was weaker than that of third-grade students. It also suggested that the school's teaching had played a role in promoting the development of their digital learning competence (Table 10).

Table 10 Values of students of different grades in each dimension of V-DLC

Discussion and conclusions

Digital learning competence, as a critical skill for vocational school students in digital learning environments, plays an indispensable role in their future learning and work. To better analyze and assess the digital learning competence of vocational school students, this study constructed evaluation indicators for digital learning competence and developed corresponding questionnaires. Analysis across 872 participants confirmed the structural soundness of the model, demonstrating good fit, reliability, and validity across diverse student populations. Through large-scale assessments and data analysis, a framework for evaluating the digital learning competence of vocational school students was established. It can serve as an assessment framework for the digital learning competence of vocational school students and support subsequent empirical research on digital learning competence.

The scale captures the multifaceted nature of digital learning competence, encompassing cognitive processing and reading, technology use, thinking skills, activity management, and will management. These dimensions collectively scaffold students' digital learning capabilities, essential for navigating the digital demands of contemporary vocational education. For the initial research question, "What are the factors and structure of V-DLC?", the investigation identified five distinct dimensions. These results not only corroborate the theoretical framework initially proposed but also enrich the understanding of digital competence within the vocational education sector, offering fresh perspectives on its application and significance. Regarding the second research question, "Is the digital learning competence questionnaire for secondary vocational school students reliable and effective?", the application of both EFA and CFA verified the reliability and structural validity of the proposed scale. This verification affirms the utility of the scale as a dependable instrument for assessing educational competencies, specifically tailored to vocational settings. The study concludes with the following findings:

The digital learning of secondary vocational school students encompasses ubiquitous learning in digital environments and involves learning activities utilizing digital tools and resources in other contexts. Digital learning competence contributes to fostering students' deep and authentic learning experiences. Digital learning competence comprises three components: abilities, goals, and will. Abilities include cognitive information processing, reading skills, technological use, and thinking skills. Goals include time management, task management, partner management, and resource management. Will is mainly reflected in willingness; it mainly refers to learning beliefs, motivation, and self-efficacy.

The digital learning competence of secondary vocational school students (V-DLC) primarily includes five closely interconnected dimensions: cognitive processing and reading, technology use, thinking skills, activity management, and will management. Cognitive Processing and Reading & Technology Use: These dimensions provide the cognitive and informational foundations necessary for conducting effective digital activities. As such, they emphasize the need for vocational education to integrate comprehensive digital literacy training that enhances both cognitive and technological skills. Future research might explore targeted educational interventions that aim to bolster these foundational skills, assessing their impact on overall learning efficacy. Thinking Skills and Activity Management: The importance of thinking skills and activity management in guiding students toward specific learning objectives underscores the need for curricula that foster critical thinking and effective management techniques. It also reflects the contemporary demands on vocational school students, as members of the digital generation, for active and self-directed learning. This approach could help students better handle the complexities of digital tasks and projects, potentially leading to improved educational outcomes. Will Management: The will component, particularly reflected in students' willingness, learning beliefs, motivation, and self-efficacy, plays a crucial role in sustaining learning behavior. This finding suggests that educational programs should not only focus on skill and knowledge acquisition but also on cultivating a motivational and self-efficacious learning environment. Investigating the effects of motivational strategies in vocational training could provide valuable insights into enhancing student engagement and persistence.

The analysis across different grade levels revealed distinct variations in digital learning competencies among vocational school students, indicating developmental progressions in their ability to effectively engage with digital tools and resources. Senior students demonstrated more advanced competencies in cognitive processing and technology use, reflecting their extended exposure to and interaction with digital environments. Conversely, younger students exhibited foundational skills but lacked the depth and proficiency observed in their older peers. This gradation underscores the importance of a tailored educational approach that progressively builds digital competencies from simpler to more complex tasks, aligning with students' developmental stages.

These findings collectively highlight the contemporary demands on vocational students as members of the digital generation, who require active and self-directed learning capabilities. There is a clear implication for educational policy and curriculum design to incorporate these elements more fully into vocational training programs, ensuring that students are well-prepared to meet the challenges of the digital age. The study enhances the understanding of digital learning competence within the context of secondary vocational education, a significantly underexplored area in existing research. Building on the foundation laid by previous studies that primarily focus on broader educational levels, this research develops a specialized framework for secondary vocational school students. The introduction of a tailored digital learning competence scale for this demographic extends theoretical constructs surrounding digital literacy and competence into the vocational realm. By doing so, it not only responds to calls for more nuanced educational tools that reflect specific learning environments but also deepens the theoretical discourse on how digital competencies are framed and assessed in vocational settings. This contributes to a more comprehensive understanding of the factors influencing digital learning success in vocational education, expanding upon the existing literature that has predominantly centered on general education or higher education scenarios.

Based on the conclusions drawn from this study, there is a compelling need for educational institutions to actively integrate digital learning competencies into vocational education. Schools should develop intelligent learning environments that support the use of advanced technological tools, enhancing students' practical engagement with digital platforms. Curriculum developers are urged to embed digital competencies within professional subjects, ensuring that students not only learn these skills theoretically but also apply them in practical, project-based scenarios. This strategy will better prepare vocational students for the digital demands of the modern workplace, fostering their ability to manage information effectively, utilize digital tools proficiently, and engage collaboratively in digital environments.

Despite its contributions, the study acknowledges certain limitations that future research could address. This study validated the stability of the five-dimensional model through a cross-group measurement equivalence test. However, the groups considered here were primarily based on random data grouping. In the future, further tests will be conducted to examine the cross-group stability of the model concerning age and gender. Additionally, the model constructed in this study was reviewed based on survey data. While this approach is suitable for inspecting group characteristics, it may not be as precise as behavioral observation and perception. In this study, convenience sampling was mainly adopted due to limited conditions, and a self-report questionnaire was used, making it impossible to conduct a test like the OECD's PISA. In the follow-up study, consideration will be given to choosing to cover more of Beijing’s administrative districts and more types of secondary vocational schools. A stratified sampling method will be considered to enhance the representativeness of samples, and qualitative data collection methods such as observation and interview will be included to supplement existing data and improve the quality of the study. Finally, multivariate statistical analysis methods can be employed to develop more profound relationships between dimensions and between dimensions and performance benchmarks within the model.

By addressing these areas, subsequent studies can enhance the robustness of the DLC framework and its applicability to diverse educational settings, ultimately fostering more effective digital learning strategies in vocational education.