Academic Performance in Medical Education During the COVID-19 Pandemic: A Scoping Review

Introduction

The COVID-19 pandemic is the most apparent cause that makes people unable to live a good lifestyle and education. Medical education has not escaped this impact by being forced to rapidly shift from traditional face-to-face learning to online formats.¹ It also gives a distinct impression on students. Some of them have negative attitudes toward online learning due to perceptions of decreased quality of education, distractions at home, poorer relationships with friends and faculty, and technical problems.^2–4 Other students had different opinions that online learning had advantages, including increased flexibility in learning, less travel time, more learning at home, and freedom to express their learning speed.^2,4,5 These two opinions illustrate that students interpret online learning in a balanced way.

During the COVID-19 pandemic, medical students faced different routines from face-to-face learning. It is mainly related to achieving aspects of clinical skills and laboratory practice. These two aspects seemed challenging to be implemented online. The academic performance resulted from students’ efforts showing the extent to which a person has achieved the goals that have been set. Academic performance was defined as the achievement of the learning process, knowledge acquisition, and job skills development.⁶

Several academic performance studies during the COVID-9 pandemic in education outside medicine showed varying results. Psychological education shows a difference in academic performance (final exam) between before and during the COVID-19 pandemic, where academic performance during the pandemic is better than face to face traditional methods evaluated using GoKoan.⁷ Findings on the business and arts streams students showed that the pre-pandemic group did worse academically than their peers during the pandemic. However, pre-COVID-19 students achieved better job readiness scores than their peers during COVID-19.⁸ Research on telecommunication engineering students found that students’ overall academic performance in remote emergencies was significantly better than in traditional face-to-face teaching.⁹ Business administration education showed no statistically significant difference in student academic performance. In addition, the unplanned and rapid move to online distance learning during the pandemic did not result in a bad learning experience as expected.¹⁰ Based on these varied findings, it is also necessary to take pictures in the world of medical education.

There were a large number of articles on the COVID-19 pandemic and its impact on academic performance in the field of medical education. It is necessary to have an overview of the studies carried out, the topics discussed, and the details of the studies carried out in this field, which will later aim to determine the priorities of the future research on this topic. The scoping review is one of the effective methods for conducting such review.¹¹ Many researchers use a scoping review to cover a broad subject that has not been comprehensively synthesized and is usually used as a precursor to systematic reviews.¹²

This scoping review aims to identify research trends in medical education focusing on medical students’ academic performance during the COVID-19 pandemic to enable available research to be mapped and summarized. Gaps in research results can be identified. It will also provide direction for future research on academic performance.

Methods

Study Design

We chose to scope review as the research design because it can assess the extent, reach and scope of the available literature, and identify key themes and factors. It provides information on new research areas to other researchers, resulting in potential practical implications and research related to academic performance during the COVID-19 pandemic on medical education. The specific method we used in the review follows the framework of Arksey and O’Malley,¹³ which consists of the following five steps:

Step 1: Identifying the Research Question

We identified research questions by conducting a bibliometric analysis. Bibliometric analysis is a method for exploring and analyzing a large amount of scientific data. Bibliometric analysis is useful for revealing trends that arise from topics or fields of research in articles and journals, exploring a comprehensive picture of certain domains in the existing literature, and obtaining new ideas for investigation. We use bibliometric analysis procedures from Donthu et al.¹⁴ We developed the questions based on population, concept, and context.¹⁵ We had three questions consisting of

1. What are the general characteristics of scoping review of academic performance during the COVID-19 pandemic on medical education?
2. Are there differences in academic performance before and during the COVID-19 pandemic on medical education?
3. What are the factors that influence academic performance during the COVID-19 pandemic on medical education?

To answer the first and second questions, we provided data including author details, year of publication, country location, research method, objectives, research situation, participants, sample size, and study design. We provided details about the independent variables, instruments for measuring academic performance, and significant findings to answer the second and third research questions.

Step 2: Identifying Relevant Studies

In this step, we created a set of inclusion and exclusion criteria to focus the scope of the review and selected databases so that the acquisition of search results would be in line with the research objectives. Inclusion criteria included full-text English-language primary articles that focus on academic performance or academic achievement during the COVID-19 pandemic, published from 01 January 2020 to 30 May 2022, containing elements of the population, concepts, and context as a format in the research process. Exclusion criteria included non-primary research articles such as all reviews, editorial letters or comments, case reports, and other documents outside the area of health and medical education and articles related to performance in patient care cases or decision-making.

The data search which was carried out by two reviewers (Y.I., M.A.), systematically examined academic performance studies during the COVID-19 pandemic in an online format. The articles were obtained using six bibliographic databases (PubMed, ProQuest, EBSCOhost, ERIC, Science Direct, Google Scholar) and PROSPERO implementing the following terms: ((COVID-19 [Title/Abstract]) AND (Academic Performance [Title/Abstract]) OR (Academic Achievement [Title/Abstract]) AND (Medical Education [Title/Abstract]) OR (Medical Student [Title/Abstract]). Our search was limited to studies on a population of medical students (medical graduates, medical professionals, and residents), English, and supplemented by a manual search for reference lists of identified papers. The PROSPERO database was evaluated using the steps described above (title/abstract section edited) to confirm that there was no systematic review scoping studies on recent or ongoing work that has been completed on the topic. The last database search was conducted on May 30, 2022.

Step 3: Selection of Research Results

We imported all titles into Mendeley Desktop reference manager software (version 1.19.8) for the title, abstract filtering, and data characterization then deleted the duplicates. Full-text articles included in the list were analyzed and evaluated independently for eligibility. We use an iterative approach to the selection of evidence sources. Two of the four authors (Y.I., M.A.) independently applied a screening tool to all titles and abstracts of retrieved articles to determine their eligibility for full article review.

To facilitate calibration, the authors met three times during the process, with the first meeting focused on creating a shared understanding of the inclusion and exclusion criteria. The next two meetings compared selected quotes and discussed differences. For the other two authors (T.J.R., S.E.), they were necessary to consult in making a mutual agreement if there were different opinions among authors. Throughout this process, one author (Y.I.) monitored each meeting to ensure and verify the accuracy of the work and contributed to the analysis of the results. Figure 1 shows the study selection process using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement.¹⁶

Figure 1 Flow diagram of the literature selection procedure. Notes: Adapted from: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg. 2010;8(5):336–341. doi:10.1016/j.ijsu.2010.02.007.16 Copyright © 2010 The Authors. Copyright © 2010 Surgical Associates Ltd. Published by Elsevier Ltd. All rights reserved. Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/legalcode).

Step 4: Data Mapping Process

The next step to carry out was data extraction and recording process. We used Arksey and O’Malley’s “descriptive-analytical” approach for data extraction, summarizing information from selected articles and recording data in Excel spreadsheet. It allowed us to analyze selected articles through a general framework. The first author (Y.I.) developed a data extraction form to enter detailed general publication data and data related to the purpose of this study.

The forms were simultaneously calibrated based on the articles analysis conducted by M.A., T.J.R., and S.E. The first author (Y.I.) performed data extraction from the included full-text articles, which were then thoroughly reviewed by another author (M.A.), and disagreements on data extraction were resolved by consensus among all authors.

Step 5: Collating, Summarizing Findings, and Reporting the Results

The data extracted in an Excel spreadsheet were then calculated in descriptive statistics to describe the review’s characteristics. To explain the authors’ reasons for conducting the scoping review and their research questions, we used thematic analysis.¹⁷ The authors started the activity of summarizing the findings and reporting the results by coding the activity to identify the relevance of the article to the three research questions that were asked at the beginning. Right after that, the authors conducted a meeting which focused on cross-checking agreement on the overall coding rationale and research questions as well as resolving any disagreements. The authors agreed to use a descriptive code namely “general characteristics” (eg, details of the author, year of publication, country location, research method, objectives, research situation, participants, sample size, study design); and theoretical codes namely “main outcome” and “secondary outcome” (eg, independent variables, instruments to measure academic performance, and significant findings). Code frequencies were summarized and presented in tabular form. Microsoft Excel 365 (Microsoft, Redmond, WA, USA) was used to facilitate descriptive analysis and summary tables.

Results

The number of articles at each stage in the flow diagram¹⁶ is presented in Figure 1. The articles relevant to the title identification and abstract review were 287 articles. Fifty-four duplicates were removed. Twenty-four articles met the inclusion criteria and were included in the final analysis.

Based on the results of the five steps of Arksey and O’Malley, this scoping study was presented in three main categories: general characteristics, the primary outcome, and secondary outcome.

General Characteristics of Included Studies

Table 1 shows academic performance of medical students during the COVID-19 pandemic studies were mostly published in 2021 (62.5%). Most studies were conducted in single medical schools (95.8%). The research designs were Cohort and Cross-sectional studies with 50% and 45.8% in total, respectively, and 4.2% case–control studies. Most of the participants involved in the studies were undergraduate students (87.5%). Only three continents were represented in this study; most of the publications were from Asian continent (South Korea, China, Indonesia, Hong Kong, Japan, Thailand, Taiwan, Jordanian, Bahrain, and Saudi Arabia); America (the United States and Mexico); and Europe (Germany, Turkey).

Table 1 Characteristics of Research Articles Related to Academic Performance and Medical Education During COVID-19

Instruments to Measure Academic Performance

The measurement of academic performance in this scoping varies greatly (see Table 2). There were six studies using GPA^18–23; nine studies using module scores^24–32; three studies implementing summative exam scores;^33–35 two studies making use of practice/performance exams^2,36; two studies employing self-reported academic performance scores^37,38; and two studies applying laboratory examination scores.^39,40

Table 2 Overview of 24 Research Articles on Academic Performance in Medical Education During COVID-19

Primary Outcome: Academic Performance in Medical Education Before and During the COVID-19 Pandemic

Table 2 shows that 11 out of 24 publication reports on the academic performance before and during the COVID-19 pandemic. All of the studies reported comparative study before and during the COVID-19 pandemic based on learning methods (online methods vs traditional methods) within single medical school setting (n=11),^{2,24,28,31–33,35–37,39,40} with cohort as the research design option (n=10).^{2,28,31–33,35–37,39,40}

Eleven studies were included in the evaluation of primary outcomes. Four studies were conducted in Asian continent.^35–37,39 Six studies were conducted in America,^{2,24,28,31–33} and one study was conducted in Europe.⁴⁰ Most studies were conducted in 2021.^{24,28,31–33,40}

Six out of 11 studies showed differences in outcome of before and during COVID-19 pandemic^{2,24,31,35,36,39} and five other studies with no difference in outcome.^{28,32,33,37,40} It means that more studies reported the influence of COVID-19 pandemic situation to academic performance. Three studies showed that medical students scored poorly during the COVID-19 pandemic^2,35,36 and the other three studies stated that students achieved higher grades during the COVID-19 pandemic.^24,31,39

Three out of 11 published studies were conducted on learning activities in the laboratory. Two out of the three studies reported significantly different outcomes of academic performance before and during COVID-19 pandemic.^36,39

Secondary Outcome: Factors Affecting Medical Students’ Academic Performances During the COVID-19 Pandemic

Table 2 shows several studies related to the influencing factors of medical students’ academic performance during COVID-19 pandemic. We divided these factors into internal and external factors. We found nine studies examining students’ internal factors that influence students’ performance such as sleep quality,²³ motivation,²² anxiety,^19,21 stress,^27,34 students’ attitude toward online classes,^18,25 e-learning style,¹⁸ and level of persistence students.²⁴ Most studies (n=8) showed a positive and significant correlation.^{18,19,22–25,27,34} Two studies evaluating the effect of anxiety to academic performance during COVID-19 pandemic reported different results. One study reported that anxiety influence academic performance,¹⁹ and the other study reported opposite result.²¹

We found 11 studies examined external factors that influenced academic performance during COVID-19 pandemic. The factors were the use of online learning media,^{28,30,32,35,39,40} changes in the schedule of anatomy laboratory learning activities,³⁶ graduation assessment system,²⁰ online instructional strategies,³⁸ students’ attendance,²⁹ e-learning educational programs.²⁶ Five studies reported correlation of the factors to academic performance.^{20,26,29,38,39}

Discussion

A scoping review of the COVID-19 pandemic response on medical education has been carried out. Both reviews focus on the development of medical education during COVID-19 pandemic.^41,42 Both of the articles also reviewed the development of assessments implemented during the COVID-19 pandemic as well as the outcomes of learning as measured by Kirkpatrick’s levels or other outcomes. However, those studies did not report a review on comparing academic performance before and during the COVID-19 pandemic and the factors that influence the performance. Therefore, we conducted this scoping review to explore the topic of academic performance before and during the COVID-19 pandemic in medical education. This type of literature review makes it possible to assess the scope of the research field and clarify the extent of the progress, reach, approach, and interest of researchers in the scope of academic performance during the COVID-19 pandemic.

Half of the articles in this scoping review used a cohort study design since (1) cohort studies help researchers to study multiple outcomes in the same exposure; (2) this method is efficient for studying the correlation between exposure and outcome; (3) exposure variables, other confounding variables, and outcomes can be measured more accurately because the respondent requires subjective assessment or memory; and (4) retrospective cohort studies can be completed quickly and relatively inexpensive compared to prospective cohort studies.⁴³

Interestingly, equal number of studies reported lower academic performance and higher academic performance during COVID-19 pandemic. Students achieved lower grades during the COVID-19 pandemic due to (1) restrictions on socializing with peers, lecturers, and faculty only in virtual setting which led to poor interactions with classmates, lecturers, and faculty. This condition is more pronounced for first-semester students, since the first semester of medical education is a difficult transition period from secondary to higher education. Many students rely on each other for support and seek help from lecturers and faculties. Therefore, the lack of direct social interaction with classmates, lecturers, and faculty adds to the overall negative experience, including its effect on motivation.^2,35 (2) Lack of practical face-to-face training during COVID-19 pandemic will potentially cause medical students unprepared to study in real clinical setting of hospitals and public health centre.² (3) Unstable Internet access. These technical barriers make it difficult for students to engage in learning and study without distraction.² (4) A sudden shift to online learning without adequate preparation, limited resources, and inadequate technological support quality of lectures and student influence teaching and learning process and obviously students’ academic performance.³⁵ (5) And limiting time and restriction of hands on for laboratory practice such as anatomy, histology, pathology, etc influence to study and take practical exams.³⁶

Good mental and academic support will help reduce the heavy burden on students during the COVID-19 pandemic. Student support and training during the COVID-19 pandemic can be a solid foundation for understanding how to deal with other pandemics in the future. Student support methods must adapt to new circumstances and environments and provide different levels of support through online and face-to-face strategies. The level of support includes (1) the development of supportive policies in a supportive environment and culture; (2) providing active support to students with pre-existing risk factors for developing mental health problems; (3) screening of the student population to find individuals who need support; (4) provision of passive support facilities for those who wish to access support. Support for online methods can be as effective as face-to-face strategies.⁴⁴

The higher achievement during the COVID-19 pandemic was due to (1) the number of students within groups in small group learning, and the arrangement of rotating practicum times were cut to meet the COVID-19 health protocol. This setting provides students opportunity to equal learning and study in undisturbed setting.³⁹ (2) Online examination might increase the possibility of cheating during examination among students. Moreover, better scores could also be due to lower stress at home when working with exams.³¹ (3) Tenacity and perseverance are important assets to face online learning conditions during COVID-19 pandemic. Those who have the assets are more likely to overcome the barriers of online learning and led to good academic performance.²⁴ (4) Furthermore, the variety of student learning styles and supporting infrastructure (eg, video recording of expert lectures, Zoom, email, YouTube video demonstrations, including games) allow students to compensate for the missing elements of face-to-face traditional teaching.²⁴

Some variables were reported to influence medical students’ academic performance during the COVID-19 pandemic. Those were sleep quality, motivation, anxiety, stress, students’ attitude toward online classes, e-learning style, student persistence level, graduation assessment system, online learning media, online instructional strategies, attendance, and e-learning educational programs.

Poor sleep quality among medical students troubles academic performance. Literature reported that 91% of the students with poor sleep quality got a GPA of 2–2.5 (P 0.001). This situation was due to sudden changes in class schedules, the use of online methods that practically make students to learn using online learning technology and changes in their independent study schedule.²³ The motivational regulation strategies affect academic performance through cognitive learning. Implementing appropriate learning activities during the COVID-19 pandemic enables students to gain knowledge, creating high motivation to obtain good academic performance.²² The learning process that was initially face-to-face then shifted to online, delayed students to submit assignment on time due to lack of learning materials that can usually be obtained from the library, and this situation triggers anxiety. This anxiety can increase the hormone cortisol secretion.¹⁹ The same episode is experienced by students who experience stress. Stress triggers memory damage by affecting the individual’s capacity to encode and retrieve information as well as excessive secretion of the hormone cortisol, causing worse long-term memory retention, feelings of difficulty, fatigue, low self-esteem, low motivation, and sleep disturbances resulting in poor academic performance.²⁷

Students who prefer face-to-face classes have a significantly higher overall score (p = 0.021) than those who prefer other methods (synchronous and asynchronous). It is due to the ease of maintaining daily learning routines (eg, group study, finding library learning resources, interacting with lecturers, etc).²⁵ Likewise, the e-learning style can predict the average test score significantly positively. Medical students’ most preferred e-learning styles are logical, independent, audio-visual, social, active, intuitive, and verbal.¹⁸

Student persistence is also a variable that affects academic performance during the COVID-19 pandemic. It is due to students’ persistence in adapting to meeting and exceeding the challenges triggered by the COVID-19 learning environment. Persistence is shown by perseverance. Higher levels of persistence are more likely to overcome the problems they face. They are confident in their learning abilities.²⁴

Limitations

Several limitations arise regarding the review process. Sources of evidence are limited to those published between 01 January 2020 and 30 May 2022, so studies in the literature published after that date cannot be reviewed which might cause present new findings and valuable information will not be evaluated. In addition, the study did not consider variations in the progression of the pandemic across countries. However, we believe that the breadth represented in this review allows for a preliminary mapping of several different contexts, terms, and possibly conceptualizations of academic performance during the COVID-19 pandemic in the medical education literatures. Furthermore, this scoping review did not discuss the assessment methods used in the assessment of academic performance, even though the assessment method can also be an influencing factor for academic performance; for example, student-peer assessment, student-self-assessment, and essays with open books. Frequent weekly testing results in somewhat higher learning gains than periodic testing; for example, only once or twice during a semester.⁴⁵

Implications for Further Research

Throughout our search, this literature shows that this scoping review is an exploratory study of the academic performance status of medical students during the COVID-19 pandemic. We suspect that many opportunities can still be investigated both from the aspect of the independent variable and the dependent variable. Aspects of independent variables to academic performance that can be studied include learning environment, student engagement, teacher, self-directed learning readiness, coping mechanism, spiritual quotient, and emotional quotient. Academic performance can be evaluated based on student satisfaction, length of the study period, career success, and achievement of learning goals.⁴⁶ The Gradient Boosted Trees algorithm is one of the appropriate measurements for academic performance.⁴⁷ Other learning contexts such as laboratory activities, clinical skills, and clinical education can also be re-examined because of the limited findings in this scoping review.

Conclusion

There is a tendency to improve academic performance by controlling variables that have a direct or indirect effect and selecting research methods that align with research objectives. The literature shows differences in academic performance before and during the COVID-19 pandemic on medical education. Academic performance during the COVID-19 pandemic was influenced by internal and external factors.

Due to the finding that online study during COVID-19 pandemic resulting both poor and good academic performance, this can be the reason for medical education institution to conduct hybrid learning (online–offline).

Future studies need to discuss other variables that can affect the academic performance of medical students during the COVID-19 pandemic such as learning environment, student engagement, teacher, self-directed learning readiness, coping mechanisms, spiritual and emotional quotient, and how the COVID-19 pandemic can affect the population in different contexts on academic performance.