On-campus living and learning presents a means of equalizing access to resources for higher education students from differing backgrounds. When universities across South Africa closed in March 2020 in order to mitigate the spread of COVID-19, many students returned home to environments where they lacked access to data and learning devices to support remote learning (Whitelaw, Culligan, & Branson, 2020; Centre for Teaching and Learning (UFS), 2020), thus increasing inequalities in student learning environments. The extent to which the move to remote teaching and learning in the university sector increased the relevance of the home environment and socioeconomic background to students’ enrolment and academic performance has not yet been quantified at scale in South Africa. This is one of the research themes in Siyaphambili’s project “The COVID-19 health crisis and inequalities in post-school education in South Africa”, which is funded by the Spencer foundation. In this article, we showcase initial results of our study on the effect of COVID-19 closures on South African undergraduate students’ academic performance.
We use annual information on students enrolled at public universities in South Africa, captured in the Higher Education Management Information System (HEMIS). The use of HEMIS data allows for a system-wide assessment of student strategies and changes in performance in 2020. We use student-level information on student dropout[1], the number of courses taken and the number passed, and further classify students by whether they receive National Student Financial Aid Scheme (NSFAS) funding or not. NSFAS information is not captured consistently across all institutions meaning that the analysis sample is restricted to 13 of the 26 institutions.
Measures of dropout and the number of courses taken allow us to assess whether students adjusted their enrolment and course load in response to the conditions they faced during 2020. We rely on the number of courses passed to assess performance. Courses passed is a crude measure of knowledge acquired and can be influenced by factors that could have changed in 2020 including content covered, teaching effectiveness, marker leniency and cheating.
In addition to student-specific factors, institutional differences are likely to have affected how the pandemic played out for students at different institutions in 2020. The South African public higher education system is intentionally diverse and differentiated in order to address a wide range of emerging skills and knowledge needs (Council on Higher Education, 2013). Institutions differ in the composition of their student bodies, their geographical location and the infrastructure and resources available in them, typically linked to their historical legacy. Together, these differences affected institutions’ abilities to facilitate remote teaching and learning. While some institutions were ready to resume the academic year remotely less than a month after closures, others experienced delays while measures were put in place to support remote delivery of learning content (e.g., zero-rating e-learning platforms).[2]
We focus on students starting their first qualification in 2018 or 2019, referred to as the 2018 cohort and the 2019 cohort, respectively. These student cohorts entered universities under the new NSFAS policy. Students from families with incomes below R350 000 are eligible for a bursary to cover the full cost of studying, including tuition, housing and a living allowance. This presents an interesting context under which to assess student responses to COVID-19. Tuition and residence-fee bursaries remove the direct cost of attendance for those funded. At the same time, students receiving NSFAS funding are, by the design of the policy, from lower socioeconomic status households. When campuses closed in 2020, students returned home regardless of their funding status. Thus vulnerability to socioeconomic hardship as well as limited data and devices for learning, would have been highest among the NSFAS-funded group.
Table 1: Institution and student summary statistics
Notes: The 2019 cohort is the cohort affected by COVID-19 in year 2. Sample restricted to South African students entering their first qualification in 2018 and 2019. Small refers to institutions with <20K students, medium 20-30K and large to >30K. Dates in the “Resume teaching” column refer to 2020.
Table 1 provides key summary information about the institutions and students in our analysis. Institutions differed in the date that they resumed teaching[3] and the online platforms used. Most traditional and comprehensive institutions resumed classes online towards the end of April or early May, with the universities of technology in our sample delaying until 1 June 2020. The table evidences institutional differences in the share of the student body funded by NSFAS, as well as the share that lived in campus accommodation before 2020. A greater share of the student bodies at comprehensive universities and universities of technology are funded by NSFAS compared to traditional universities, but there is substantial heterogeneity with respect to the share of the student body in campus accommodation both across and within institution types.
Figure 1: Estimated dropout rates by cohort and NSFAS funding status
Notes: Estimate from a Probit model controlling for institution, qualification type, female and share of courses passed in year 1.
Figure 1 compares the estimated dropout rates on entering second year for our two cohorts, separately for students funded by NSFAS, versus those not funded by NSFAS. The figure shows an increase in the estimated percentage dropping out between the 2018 and 2019 cohorts, but the increase is larger, and only significant, for those not on NSFAS funding: an increase of 20% compared to 6% among the NSFAS funded group.
Next, we assess whether students adjusted the number of courses that they took and/or had differential performance in 2020. The 2018 cohorts’ second-year outcomes act as a counterfactual for the 2019 cohorts’ second-year outcomes. Looking at these trends in isolation, however, conceals the fact that the 2019 cohort was passing more courses in their first year than the 2018 cohort in their first year. We therefore employ a difference-in-differences approach, which allows us to separate the effect of the COVID-19 pandemic from potential time trends in student performance.
We find no change in the number of courses taken in 2020, regardless of funding status, suggesting that students did not adjust their course load during 2020. After controlling for time trends, average cohort performance in year 1 and number of courses taken, we found changes in the number of courses passed in 2020 at 10[4] of the 13 institutions. Students in second year at RU, SPU and CPUT passed a similar number of courses in 2020 to the prior cohort.[5] Students not funded by NSFAS in the institutions where we found a change, passed between 0.22 (SU) and 1.33 (MUT) additional courses in 2020. Seven[6] of the 10 institutional regressions find evidence that improvements in the number of courses passed were lower among students on NSFAS funding. With the exception of UCT, however, the total 2020 impact for NSFAS students was still positive, albeit smaller and not always statistically significantly different from zero.
While there are nuances at different institutions, this overall impact for the sector equates to approximately two fifths of a course extra passed over the year, with NSFAS funded students gaining about a third of an additional course.
We recognise that although we can broadly attribute changes to “COVID-19” using this approach, there were a multitude of factors impacting students during this time. The data does not allow us to directly identify the different pathways through which student performance may have been affected. Our analysis of the HEMIS data suggests that NSFAS funding may have provided students with a safety net during a time of great uncertainty, preventing increases in dropout evidenced for students not on NSFAS funding. Among students remaining enrolled, NSFAS-funded students’ performance increased on average (in terms of courses passed) but by less than the corresponding increase among students not on NSFAS. This difference could be explained by NSFAS-funded students experiencing worse learning conditions in 2020, but could equally be attributed to lower rates of cheating, for example. Nonetheless, since students’ home background is not explicitly observable, factors such as marker leniency, which have been anecdotally suggested to contribute to improved student performance in 2020, are unlikely to have affected students from different socio-economic backgrounds differently.
The results showcased in this article lay the groundwork for a more in-depth national analysis of institutional changes in South Africa’s university sector stemming from COVID-19. The study builds on our detailed analysis of student performance with respect to credits passed at the University of Cape Town (UCT), using data provided directly by the institution.[7] This data includes information on students’ prior grade point average (GPA) as well as their performance in 2021. This means that we can explore whether students at different points on the academic performance distribution experienced changes in their outcomes in 2020 differently, as well as observe whether students’ academic trajectories suffered, improved, or stayed the same in 2021.
The UCT study shows that while the achievement gap between NSFAS and non-NSFAS students (in terms of credits passed) did not change significantly in 2020, by the end of 2021 achievement gaps had grown between students in the bottom quartile of the pre-COVID-19 GPA distribution, and to an even greater extent once differential dropout is recognised. These results also reveal that overall performance gains in 2020 were concentrated among the bottom quartile of the pre-COVID-19 GPA distribution, with this gain reversing in 2021. This is suggestive that performance improvements were not true learning gains, in the sense that they did not translate into sustained improvements.
References
Centre for Teaching and Learning (UFS). (2020). UCT SAULM Data Summary. Department of Higher Education and Training.
Council on Higher Education (South Africa). (2013). The Higher Education Qualifications Sub-Framework. OCLC: 881681732. Pretoria: Council on Higher Education (CHE).
Department of Higher Education and Training. (2022). Draft Policy for the Recognition of South African Higher Education Institutional Types. Available:
Whitelaw, E., Culligan, S., & Branson, N. (2020). Student ability to learn at home. An introductory look at student access to remote learning resources. Southern Africa Labour and Development Research Unit, University of Cape Town.
Data
Department of Higher Education and Training. Higher Education Management Information System, 2012 to 2020 [data set]. Pretoria: Department of Higher Education and Training [producer]. [privately distributed].
[1] Students who did not re-enrol at the same institution in the year after starting their qualification and who had not graduated.
[2] See Universities South Africa’s summary on remote learning readiness here.
[3] Institutions also had different timeline for return to in-person teaching and some institutions extended the academic year to include initial months of 2021.
[4] UKZN, WITS, NWU, UCT, SU, UZ, UJ, NMU, MUT, TSUT.
[5] Dropout rates were also found not to have increased significantly during 2020 for students at RU and SPU.
[6] UKZN, WITS, UCT, UZ, NMU, MUT, TSUT.
[7] See Whitelaw, E., Branson, N., Leibbrandt, M. (2022). Learning in lockdown: University students’ academic performance during COVID-19 closures. Cape Town: Southern Africa Labour and Development Research Unit, University of Cape Town. (SALDRU Working Paper Number 289).