Assessing the significance of course materials for graduate program admission

Abstract

Enrollment into graduate programs continue to increase. Given major changes occurring within education (e.g., online courses and OER integration), it is imperative for faculty and students to be aware of the impact these changes could entail to those with higher education aspirations. The use of Open Educational Resources (OER) is one important change that has occurred with respect to the types of academic materials used in higher education courses. The use of OER has significantly expanded over the past decade. Resources accessible under the caption of OER are free and available in the public domain. Currently, there is little information regarding how these changes might affect graduate school applications or admission. The purpose of this study was to examine the importance of academic materials used in undergraduate classes in the evaluation of individuals for graduate admission. The results indicated that academic materials are the least important factor utilized for admission criteria. This result may indicate that the use of Open Educational Resources, low cost or free course materials with a copyright license that allows for reuse, should not impact the likelihood of a student being accepted into a graduate program.

According to McFarland et al. (2019), the number of students attending colleges and universities reached 19.9 million in Fall 2018, which is a significant increase from 15.3 million higher education attendees in 2000. From those 19.9 million, some 2.9 million are expected to enroll in post-baccalaureate programs (McFarland et al., 2019). Graduate program application itself has been steadily increasing at an average annual rate of 4 % between Fall 2007 and Fall 2017, and it is projected that post-baccalaureate enrollment will increase by 3 % annually between 2017 and 2028 (McFarland et al., 2019).

In addition to the changing dynamics of the student body, other transformations occurring in higher education are tied to improvement and propagation of new initiatives. One example is the advent of Open Education Resources (OER), which has contributed significantly to cost effectiveness of the course materials (Wiley, Green, & Soares, 2012). OER are defined as teaching, learning, and research resources that reside in the public domain or have been released under an open intellectual property license that permits their free use and re-purposing by others (Olcott Jr, 2012). There are two important dimensions to openness pertinent to OER initiatives. The first is related to the free availability and accessibility of the material over the Internet, and the second dimension relates to the level of restrictions on the use of the resource. In order for material to fall under the caption of OER, technology and price should not be barriers for use, and legal permission barriers should be minimized for the consumer. With the use of author attribution, the end-user should be able to reuse, adapt, revise, or remix the materials (Hylén, 2006).

OER initiatives have potential to address challenges in course material costs that both students and faculty face. U. S Public Interest Research Group estimates that college students spend an average estimate of $1200 per year on textbooks and other supplies, which is 14% of the cost of tuition in at a four-year public university. Since 1994, college textbook prices have increased nearly four times the rate of inflation, which makes it a significant factor fueling the rise of costs in higher education in general (Senack, 2014). OER materials are important tools which have the potential to significantly reduce the cost of educational materials and to address financial barriers to student academic success (Hilton III & Wiley, 2011). The most common factor cited by faculty when selecting educational resources was the cost to the students, and OER materials are undeniably a less costly alternative to traditional course materials used (Allen & Seaman, 2016). A survey conducted by Allen and Seaman (2016) of 3000 U. S faculty reported that 5.3% are already utilizing openly licensed textbooks. Additionally, a significant increase in awareness was noted regarding OER materials. As more students attend college, more are facing a potential financial barrier (De los Arcos, Farrow, Perryman, Pitt, & Weller, 2014). The adoption of OER by faculty could potentially lead to dismantling this financial barrier for students at all levels of higher education.

Although some studies have reported positive outcomes in faculty awareness and interest in adopting OER, barriers to widespread OER adoption still exist (Belikov & Bodily, 2016). A majority of concerns expressed by faculty stem from the quality of OER. Allen and Seaman (2012) found that the concern reported most frequently was that no one clearinghouse exists to ensure quality OER materials are available. Other studies indicate that this concern remains strong amongst faculty and even some students, leading faculty to favor traditional textbooks (Allen & Seaman, 2012; Belikov & Bodily, 2016; Jung, Bauer, & Heaps, 2017; Watson, Domizi, & Clouser, 2017). Specific reasons for favoring traditional materials over OER course materials include associating free with a lack of quality (Belikov & Bodily, 2016), a lack of time to evaluate the quality and match content to the course (Belikov & Bodily, 2016; Jung et al., 2017; Watson et al., 2017), and even a fear of perceptions of other faculty members regarding competence as a professor (Watson et al., 2017). In some cases (19% of respondents), faculty even report beliefs that OER are inferior to traditional textbooks in their quality and ability to prepare students to succeed in their future courses (Jung et al., 2017). Faculty reported their perception of quality was significantly impacted by OER lacking essential materials, the use of outdated research or teaching methods, poor editing quality, poor illustrations, the lack of supplemental and multimedia materials, and alignment of content and user’s need (Jung et al., 2017). As noted by both Jung et al. and Watson et al., when faculty have exposure to and experience with OER they tend to adopt more favorable views of OER materials. Jung et al. (2017) found that when faculty used OER materials for their course, they “perceived their students as equally prepared for their course using open textbooks compared to using traditional textbooks” (p. 131), and an additional 20% of respondents reported the effects on student preparedness to be superior to traditional textbooks.

While the positive change in faculty attitudes towards OER materials is observed after material adoption, the faculty that currently hold negative views on OER may not seek the opportunity to use them in order to see the benefits. Allen and Seaman (2016) found that lower levels of graduate and specialized programs compared to other degree types reported favorable views of and willingness to use OER materials in their courses. Therefore, it appears that graduate programs may be least likely to adopt OER materials. Allen and Seaman (2012) found that when reporting on the value of OER, baccalaureate, masters and doctoral degrees to have the lowest percentage in ascending order with doctoral degrees reporting only 15% of doctoral faculty found them valuable.

Current research has focused on more general views assessing faculty perceptions, barriers and incentives to adopting OER materials, but very little research has assessed the impact of OER materials on various student outcomes. One of the areas yet to be addressed is whether a widespread program adoption of OER materials will impact a student’s chance at being accepted into a graduate program. As noted earlier, student interest and enrollment in graduate programs has been steadily increasing. Will the increase in adoption of OER materials among colleges change the landscape of how graduate admissions assess student preparedness for their programs?

Accurately forecasting which students are best suited for a given graduate program is imperative for both students and graduate programs. Due to that, admission procedures play crucial role in determining who has access to training, opportunities and other benefits available at the graduate level. Factors determining admission decisions vary according to higher education level. Grades in high school is top decision factor for first-time freshmen student, followed by essays or writing samples, test scores, recommendations, class rank, and other considerations (Hawkins & Lautz, 2005). Conversely, admissions to graduate school has been commonly anchored around three critical application materials: standardized test scores, academic performance, often measured by GPA, and letters of recommendation (Evans, 2017; Forrest & Naremore, 1998; Galang, Yuan, Lee, & Sukotjo, 2011; Keith-Spiegel, Tabachnick, & Spiegel, 1994; Kuncel, Kochevar, & Ones, 2014; Lizares, Rahnema, Pang-Rey, Suan, & Bautista, 2016; Weiner, 2017). Student’s academic transcript also seems to play a major role in admission decision for Master’s programs, given that 99% of reviewers rely heavily on the academic transcript to evaluating past performance of the student (Okahana, Augustine, & Zhou, 2018). Evidence of utility of standardized tests, GPA and letters of recommendations in predicting student outcomes in graduate programs is available in research literature (Kuncel & Hezlett, 2007; Walters, Kyllonen, & Plante, 2006; Westrick, 2017). Nonetheless, some researchers have pointed out that the heavy reliance on these primary criteria for admission favors some ethnic groups, and as a result, increasing interest has been noted in broadening criteria for admission with an emphasis on non-cognitive constructs (Niessen & Meijer, 2017). Different non-cognitive attributes have been preferred depending on the type of program one is evaluating a student for. For instance, research-focused master’s programs prefer attributes such as creativity or curiosity, working in teams, recovery from failure, and time management. While professionally-focused master’s programs emphasized importance of crisis management, ability to work with different people and cultures, and an aptitude for service to others (Okahana et al., 2018). Additional criterion used in the selection of graduate students are referred to as second-order criteria (Keith-Spiegel et al., 1994). Although a few common themes can be found within the literature on what second-order criteria are for individual disciplines, most fields require unique quantitative and qualitative information from their candidates. Common themes of second-order criteria are related to experience within the field as well as “Good Match” factors between students, the program, and faculty (Galang et al., 2011; Keith-Spiegel et al., 1994; Landrum & Clark, 2005; Lizares et al., 2016; Weiner, 2017). Still other fields indicate that the undergraduate program or institution reputation and rigor also impact their decision to admit an applicant (Forrest & Naremore, 1998). One major gap in the literature is the impact of academic materials used in the undergraduate program on graduate student admission which, by extension, would also include the impact of OER used as primary course materials.

With research indicating that the major concerns regarding OER are related to the quality of materials, it is unclear how an admissions committee might perceive a student who attended a university that makes use of OER. Will students be viewed as attending a college that is not rigorous enough due to the use of OER materials in their courses? The intent of the present study is to explore current admissions criteria to graduate programs with a focus on the extent to which academic materials are utilized as a secondary-order criterion in the admission process. Academic materials in this study are defined as any teaching aids that are utilized by instructors to impart knowledge to their students. Academic materials come in variety of print, audio, visual, electronic, interactive, and other forms. A qualitative study was conducted to determine the answers to the following research questions:

• 1. What criteria are important when determining graduate student admission?

• 2. Do academic materials play a significant role in the decision process?

• 3. If academic materials play a significant role in the graduate admission decision-making process, what impact will OER have on that decision?

Method

Participants

This study included a total of 26 participants. The only demographic information collected was simply whether the individual responding to the survey played a role in graduate admission decisions or not. Those recruited were identified graduate program coordinators from seven public universities in the state of Minnesota. Institutional research board permission was obtained from each university. Initially, 148 graduate program coordinators were sent an email that contained a recruitment statement along with an anonymous link to the survey. A second reminder email was sent out to remind those who may have been interested in participating but have not yet had the chance to provide their responses. A total of 49 responses were collected. However, four of those 49 indicated they were not responsible for making admissions decisions, therefore the responses from those participants were excluded. Nineteen additional responses were excluded for responding to less than 80% of the survey. This left 26 participants out of the 148 recruited (82% attrition).

Measure

An online survey (Additional file 1) was developed for the current study using Qualtrics. The survey was constructed with three sections. First, participants were asked to indicate whether they were responsible for graduate admission decisions. Next, they were asked to rank a series of 13 different possible criteria they may use when considering students to be admitted to their program. For example, whether-or-not a candidate’s previous degree was, or will be, from an accredited college or university. They were asked to rank each based on their importance or weight they held in each candidate’s admittance. The 12 variables were chosen by the researchers based on a review of literature on admissions criteria in various fields of study. Variables that were most common between the studies were chosen for this section of the survey. For example, many studies indicated variables that demonstrate a student has skills that signify an ability to succeed, which can be evidenced by GPA, for academic performance; standardized test scores, for qualitative skills, quantitative skills, and communication and subject specific skills; and letters of recommendation, to further support strengths in areas assessed by standardized test scores and other relevant information (Evans, 2017; Forrest & Naremore, 1998; Galang et al., 2011; Keith-Spiegel et al., 1994; Kuncel et al., 2014; Lizares et al., 2016; Weiner, 2017). Those three variables were reported to be the most commonly requested student data that assists decision makers in understanding a prospective student’s skills, interests, and experience in and exposure to the field. Other secondary pieces of information reported as being of interest to decision makers were related to community service (Okahana et al., 2018), understanding and experience in diversity (Okahana et al., 2018), as well as the university they attended and faculty with whom they interacted (Forrest & Naremore, 1998; Galang et al., 2011; Keith-Spiegel et al., 1994; Landrum & Clark, 2005; Lizares et al., 2016; Okahana et al., 2018; Weiner, 2017). The 13th variable in the survey was added to assess the quality of academic materials in relation to other important and general admission criteria. Although the focus of this study was to determine if academic course material directly impacted graduate admission decisions, the survey questions do not make that a point of focus in order to minimize the bias this might create for or against it as a criteria. Instead, the questions ask about the academic rigor of the syllabi and the quality of academic materials in major classes. The researchers did not categorize OER separately from Academic Materials for the purpose of this study and chose not to specifically name OER as a criteria or example to promote the equivalency of OER to course material. Following this exercise, participants encountered two open-ended questions to report additional, important criteria that were not included in the provided ranking and other factors that might negatively impact candidates.

No previous reliability or validity data was available for this survey. The survey was reviewed by qualitative methodology experts and the format was revised for readability. The qualitative experts judged the survey to be valid in terms of convergence validity based on the previously conducted studies in discipline specific contexts (Forrest & Naremore, 1998; Galang et al., 2011; Keith-Spiegel et al., 1994; Landrum & Clark, 2005; Lizares et al., 2016; Okahana et al., 2018; Weiner, 2017).

Qualitative analysis

The two open-ended questions posed were assessed using a thematic analysis approach. The type of thematic analysis used was purely deductive and descriptive in nature. The goal of the analysis was to only use the words included in participant responses to create themes by grouping related words or phrases. Two levels were identified. The first level is the overarching themes. The second level is composed of smaller bits of information included in responses that together make up the larger themes. For example, various standardized tests were identified as criteria when choosing candidates. Some participants indicated that “standardized test scores” were of importance while others named specific standardized tests (e.g., MAT and GRE). Due to specific standardized scores being named that are related by the type of assessment and unrelated by specific test, the first level identified is “standardized tests” and the second level is “GRE” and “MAT”. Some only identified one level rather than two. Two were only included when the second levels were related but dissimilar enough to warrant a unique explanation. For example, some participants highlighted the importance of letters of recommendation. This was the only specificity provided in the open-ended responses, so this theme of “Letters of Recommendation” only included the first level theme.

Results

Importance of admissions criteria

The survey’s ranked data were analyzed using descriptive statistics (e.g., M, frequency and SD). Kendall’s Coefficient W was utilized to determine concordance among the faculty’s rankings. As shown in Table 1, the results of descriptive statistics indicated that faculty ranked Graduation from an accredited undergraduate program as the most salient criteria for admission, followed by Evidence of quality academic writing, Evidence of exposure to empirical research or literature. Faculty believed Quality of academic materials in major classes and Academic rigor of major course syllabi were of equal yet lesser importance, relative to other criteria, in determining graduate admission. They also ranked Publication record of the professor in the student’s major classes as the least important criteria.

Table 1 Ratings of the importance of various criteria in graduate admission

A Kendall’s W correlation was run to determine the concordance among 26 university faculty on the ranking of the criteria for graduate admission. There was a moderate, positive correlation among faculty rankings, which reaches statistical significance (W = .44, p = .00). These results indicated 44% agreement among faculty on what criteria are important for graduate admission, which can be viewed in Table 2. The researchers are cautious, however, of drawing conclusions about the statistical significance of the quantitative data due to the low number of participants and the likelihood that this may have falsely inflated the statistical significance.

Table 2 Concordance among faculty rankings

Through the qualitative analysis, nine themes with fifteen subthemes were identified for the first question and eight themes with twelve subthemes for the second question. All themes and subthemes can be viewed in addition to their frequency of appearances in the responses in Tables 3 and 4, respectively. In response to the first open-ended question about which additional variables are considered when choosing graduate candidates, the top three variables identified were academic performance evidenced by GPA, experience related and non-related to the field of study, and standardized test scores. This response specifically prompted participants to report variables that are important but not previously mentioned. However, few responses provided were different from the ranking items. The unique responses that were provided are the Fit Between Programmatic Variables, such as research interests, goals, and degree type, an Ability to Work as a Team Member, Statement of Purpose, and Letters of Recommendation.

Table 3 Additional criteria identified through qualitative analysis

Table 4 Criteria indicating applicant is not a “good fit” through qualitative analysis

The second open-ended question inquired about the elements that may indicate a candidate is not a “good fit” for the program. The top two variables identified were poor academic achievement and poor writing skills. A majority of responses reflected what participants report to be important criteria while also indicating that not meeting these criteria means candidates would not be a good fit. However, novel variables were introduced in response to this question as well. Specifically, participants first mention here that a Lack of Professionalism, such as professional Boundaries, Professional Representation of Self, and Professionalism in General; Lack of Quality Communication Skills, in general and specific to English speaking skills; and a Lack of Understanding of the Field candidates are applying to.

Discussion

Accurate assessment about admission criteria is important to both student and faculty alike. Overall, it appears that quality of academic materials and the rigor of course syllabi are of little to no concern in admission decisions for graduate students, and trends for materials that are most critical to an admissions committee are largely the same (Evans, 2017; Forrest & Naremore, 1998; Galang et al., 2011; Keith-Spiegel et al., 1994; Kuncel et al., 2014; Lizares et al., 2016; Weiner, 2017). Students who have inaccurate view on admission criteria may be at a disadvantage if they fail to acquire desired experiences valued by graduate programs. On the other hand, faculty who are aware of the graduate admission criteria have greater possibility of assisting their students in obtaining graduate enrollment. This study was primarily directed to assess graduate admission criteria used by public universities in Minnesota. Moreover, important emphasis was placed on the impact of academic materials on graduate admission. Researchers placed emphasis on assessing impact of academic materials due to increased use of alternative course materials, such as OERs, on campuses across the nation.

The indication that academic materials are not heavily weighted elements for graduate program admission is useful for the expanding use of OER in undergraduate programs. While other considerations for the use of OER must be taken into account, such as quality, current material, availability of desired topic, or student access to technology, results of this study indicate that the impact of OER use on student admission to graduate programs may be negligible (Allen & Seaman, 2012; Belikov & Bodily, 2016; Jung et al., 2017; Watson et al., 2017).

Because the agreement among the faculty regarding admission criteria was moderate, it is important to emphasize the distinctiveness among programs in utilizing admission criteria. Nonetheless, the results also highlight the relative importance of graduating from an accredited undergraduate program, a student’s quality of academic writing, one’s exposure to empirical literature, applied research experiences, and diversity and cultural experiences. Faculty and advisors may want to discuss the academic experience relative to their discipline, such as academic writing or research experience, so that their students are aware of its importance when applying for a graduate program. Other areas faculty and advisors may want to discuss with their students are experience in their desired field of study, academic writing and communication skills, attending to the mission or goals of the program and faculty, and writing a high-quality personal statement in addition to primary-order criteria, such as GPA, standardized test scores, and letters of recommendation. Students would benefit greatly from a push to develop experiences and skills in these areas as graduate admissions coordinators report them as other important variables in multiple places throughout the survey and across many fields of study.

Limitations

Although this study provides insight to the current ways graduate coordinators assess student candidates for their programs, this study is not without its limitations. First, the impact of academic materials from a student’s current or previous college program was assessed, but this survey did not explicitly ask participants their perceptions of OER materials. The purpose for not directly asking about OER materials versus traditional textbooks was to keep the subject of OER from drawing focus away from the main point of inquiry – namely, whether textbooks or other course materials a factor in graduate admission. However, this also limits the researchers from drawing a direct conclusion that OER materials do not impact an admission decision. Researchers can, however, still determine that academic course materials are not of large concern by most graduate programs.

A second limitation is the generalizability of the study results. The results from this study are limited to not only the state of Minnesota, but also to public universities since only participants from public universities in the state of Minnesota were recruited for this study. Therefore, any conclusions drawn cannot be extended to private or for-profit colleges. Additionally, a majority of programs represented in this study are primarily in-person programs, many with limited online course offerings as the state of Minnesota has approximately 30 online graduate programs out of 116 offered (Minnesota State Colleges and Universities, 2019). No solely online graduate programs can be assumed to hold the same perceptions as those schools reported in this study.

Future research in this area should continue to assess perspectives of OER materials and the impact on graduate admissions. As it becomes increasingly popular for OER materials to be adopted by colleges, it is important to continue assessing perceptions with this trend. Perhaps the graduate programs at Minnesota public universities have a limited awareness and understanding of OER materials, at least at the graduate level. Should this be the case, an increase in awareness and understanding of OER materials and other course materials may change the impact they have on graduate admissions criteria. Another potential area of investigation is the actual impact of preparedness for graduate programs on students graduating from universities using OER. It is important to know not only whether students will be given the opportunity to attend a graduate program based on the university’s and faculty’s choice of course materials, but also to understand whether course materials will lead to student success once admitted to the programs.

Because the alternative forms of academic materials such as OER are on the rise and their use has shown to reduce costs for students significantly, it is important to determine the impact of their use on graduate admission. The findings of the study indicate that academic course materials are among least important variables used to make admission decision. This finding is significant, because it allows us to infer that the use of alternative course materials might not have a significant impact on students’ odds for graduate enrollment. Further research is needed in order to determine specific impacts of the use of alternative course materials.