The complexities of evidence-based policymaking

Image: Qimono on Pixabay.

How should policy decisions be made in a world where there is always some degree of uncertainty? In a note leaked in January this year preceding the State of the Nation Address, the Presidential Economic Advisory Council (PEAC) put forward several provocative views about a way forward for economic policy in the country. Many of the PEAC recommendations do not have a clear evidentiary basis, but they are at least accompanied by calls for impact assessments. Nonetheless, past experience in South Africa and abroad shows that “impact assessments” are not sufficient for ideal evidence-based policymaking – a credible policy review process requires also critically examining the quality of the evidence that is being considered, as well as considering any potential perverse incentives amongst both researchers and policymakers. In this article we take the opportunity presented by the PEAC note to explore some systemic reasons for challenges in the research-policy relationship, using the Employment Tax Incentive (ETI) as an instructive case study.

What kind of evidence?
Impact assessments of the kind referred to in the PEAC note typically comprises quantitative data analyses that use either randomised control trials (RCTs) or observational “programme evaluation”-style econometric techniques. These methods can be very powerful, providing rigorous and highly credible evidence, and the 2019 and 2021 Nobel Memorial Prizes in Economic Sciences were awarded for advances in these areas. However, in order to use these methods appropriately, one needs to be cognizant of the societal and institutional contexts, the specific questions of interest, and how the best practices in these methods are constantly evolving. A failure to apply due care, for example by using simple cookie-cutter applications of these methods, can lead to poor quality research which at best will be uninformative and at worst can lead to unsuitable and very costly policy decisions. Not all evaluations are equal.

Misinterpreting the evidence
In addition to the aforementioned concerns about the validity of a particular study’s findings, there are some systemic reasons that explain why the results of such impact assessments may be misinterpreted.

Perhaps most obviously, people who have publicly supported a policy have some interest in showing that these policies “work”. This may cause them to disregard problems with studies which have favourable results, while downplaying the importance or validity of studies which have unfavourable results.

The difficulty of saying “we don’t know”
Researchers have some perverse incentives too. They’re typically under pressure to produce substantive and interesting results, and research incentives are heavily geared against simply concluding that the evidence is insufficient or that the technique failed, and “we don’t know” whether the policy worked. To avoid this kind of conclusion researchers may paper over or even omit tests of critical assumptions, or subtly change the question of interest. More technically, researchers can manipulate the data by evaluating endless subsamples and outcomes until by chance they find and report “significant” results, even though these are very likely to be statistical false positives (see Brodeur et al, 2020; or xkcd for lighter reading).

Policy relevance and policy recommendations
Relatedly, and because funders sometimes explicitly require this, researchers are also incentivized to emphasize the “policy relevance” of their work. This can encourage researchers to make unfounded policy recommendations. Even the results from a high-quality study must be weighed against the quality-adjusted body of evidence and other policy-relevant considerations. A poignant example from India’s public employment programme was highlighted recently, where academics promoted an electronic payment system because it may have reduced corruption — but did not consider how it also massively increased the waiting times for wage payments (Dreze 2022). Good policymaking entails criteria and objectives that are broader than those used to evaluate academic work, and policy prescriptions based on one key academic finding may be harmful in other dimensions.

The Employment Tax Incentive as a case in point
In the South African context, the way academic evidence has been used to institute and expand the Employment Tax Incentive (ETI) is illustrative of many of these issues. The ETI is probably the most high-profile active labour market policy of the last decade, although it has been politically contentious since its inception.

Policy adoption
The ETI was initially instituted on the back of an academic RCT, which subsidised the employment of a random subset of young people. The main conclusion of the paper was that the subsidy worked, as those youth who received the subsidy were more likely to be employed than those who didn’t.

While this may seem to be straightforward evidence that instituting the ETI would be a good idea, the study actually answered a subtly different question than what was needed to justify the policy. As the authors carefully note, the RCT was materially different from the ETI policy, and did not evaluate whether the subsidy actually created employment, which is the objective of the ETI (Levinsohn et al, 2014). Instead, it showed that firms prefer subsidised young workers to non-subsidised young workers – which does not imply that firms created new jobs to employ subsidised workers.

However, as Muller (2021) notes, the RCT’s results were nonetheless used to promote the ETI in the run-up to its adoption (also see Muller 2021 for another general discussion of the ETI as illustrative of challenges in the research-policy relationship). Even aside from any implementation or technical issues (e.g. few firms actually claimed the subsidy), the RCT was conceptually not fit for the purpose of evaluating the prospects of the ETI, and yet it was key evidence cited in favour of adopting the policy.

Policy expansion
Expansions and extensions of the ETI since then have also made frequent recourse to evidence which says that the policy creates jobs –the recent PEAC note is another such example. And yet the reality is that the existing evidence is deeply contradictory and divided.

Studies which compare ETI-eligible individuals (e.g. young people) to non-eligible individuals (e.g. old people) have consistently found no employment effects of the policy, and in some cases very precise null results (Ranchhod and Finn, 2015;2016; Ebrahim, 2020a*).

In contrast, work which compares employment at ETI-claiming firms to employment at non-claiming firms finds increased employment at the ETI-claiming firms (Ebrahim 2020b**; Bhorat et al, 2020).

The difficulty with firm-level approaches is that firms which claim the ETI are quite different from non-ETI firms, even before the ETI comes into effect (this is called a selection bias). Among other things, ETI firms are bigger and they grow faster before ETI take-up. Therefore, when researchers compare employment growth across these two types of firms, the difference may very well not be due to the ETI, but may simply be reflecting pre-existing differences in the firms – thus confounding any estimate of a causal effect of the ETI, and very likely leading to false positives. It is crucial that we “compare apples with apples” when evaluating policies of this sort.

The firm-level papers implement methods to address this, but Budlender & Ebrahim (2021) show that the assumptions required for these methods likely fail.

Technical points aside — and this is not an exhaustive review of the potential technical issues — the overarching point is that it is simply incorrect to state that “the evidence” shows that the ETI creates jobs, despite frequent assertions of this type. In fact, the only impact assessment of the ETI which has been published in a peer-reviewed journal thus far is one of those which finds no effect of the policy (Ranchhod & Finn, 2016).

Even a brief review of the ETI debate shows a selective reading of the evidence, misplaced claims to policy-relevance, and that research quality matters.

These issues and others ultimately add noise to the literature. Evidence needs to be read comprehensively and with attention to quality, both of which take time and considerable training. Overall, “evidence-based policy” can actually be a minefield for policymakers, not least of all when researchers claiming to review the evidence overstate the confidence in their recommendations.

We do not know with certainty whether the ETI creates jobs or not. And in an environment where there are few alternatives and the policy challenges are overwhelming and urgent, policymakers may very well be justified in taking a risk in implementing and expanding the ETI despite unclear evidence of its efficacy. But in this case we should be transparent about what the evidence does say.

We also do not know what specific changes will improve the systemic issues that we have highlighted in the research-policy relationship. At a more general level, however, greater transparency and some form of institutionalised peer review of policy recommendations would probably be helpful practices. At the very least, we hope that a broader awareness of these issues is valuable in and of itself.


* The earliest version of Ebrahim (2020a) was first published as Ebrahim and Pirttilä (2019). Ebrahim (2020a) is a PhD chapter, which we cite as the latest version of this work.
** The earliest version of Ebrahim (2020b) was first published as Ebrahim et al. (2017). Ebrahim (2020b) is a PhD chapter, which we cite as the latest version of this work.


Peer-reviewed references
Brodeur, A., Cook, N. and Heyes, A. 2020. Methods Matter: p-Hacking and Publication Bias in Casual Analysis in Economics. American Economic Review, 110(11): 3634-60.

Muller, S. 2021. Evidence for a YETI? A Cautionary Tale from South Africa’s Youth Employment Tax Incentive. Development and Change, 52(6):1-42.

Ranchhod, V. and Finn, A. 2016. Estimating the Short Run Effects of South Africa’s Employment Tax Incentive on Youth Employment Probabilities using A Difference-in-Differences Approach. South African Journal of Economics, 84(2):199-216.

Non-peer-reviewed references
Bhorat, H., Hill, R., Khan, S., Lilenstein, K. and Stanwix, B. 2020. “The Employment Tax Incentive Scheme in South Africa: An Impact Assessment”. DPRU Working Paper 202007, Development Policy Research Unit.

Budlender, J. and Ebrahim, A. 2021. “Estimating Employment Responses to South Africa’s Employment Tax Incentive”. SA-TIED Working Paper No. 187, UNU-WIDER.

Dreze, J. 2022. “On the Perils of Embedded Experiments”. Ideas for India: Perspectives. 10 March.

Ebrahim, A. (2020a). “A policy for the jobless youth: The Employment Tax Incentive”, chapter 5, pages 112-146. University of Cape Town. PhD Thesis.

Ebrahim, A. (2020b). “A policy for the jobless youth: The Employment Tax Incentive”, chapter 4, pages 85-111. University of Cape Town. PhD Thesis.

Ebrahim, A., Leibbrandt, M. and Ranchhod, V. (2017). “The effects of the employment tax incentive on South African employment”. WIDER Working Paper 2017/5, UNU-WIDER.

Ebrahim, A. and Pirttilä, J. (2019). “Can a wage subsidy system help reduce 50 per cent youth unemployment: Evidence from South Africa”. WIDER Working Paper 2019/28, UNU-WIDER.

Levinsohn, J., Rankin, N., Roberts, G. and Schoer, V. 2014. “Wage Subsidies and Youth Employment in South Africa: Evidence from a Randomized Control Trial”. Stellenbosch Economic Working Papers 02/14. Bureau for Economic Research.

Ranchhod, V. and Finn, A. (2015). “Estimating the effects of South Africa’s youth employment tax incentive-an update”. SALDRU Working Paper 152, Southern Africa Labour and Development Research Unit.