This blog is based on my 2017 PD Leake lecture at the ICAEW on the challenges and opportunities for accounting and financial markets research when it comes to evidence-based policy making. The full paper is available here.
Evidence-based policymaking is a rigorous attempt to base policy decisions (e.g., new regulation) on scientific and empirical evidence, including impact studies, cost-benefit analyses, program evaluation, etc. It broadly captures the idea that science and empirical evidence are used rigorously and comprehensively to inform policy decisions. This idea is very appealing. Policy making based on empirical evidence and sound theory should lead to better policies: research can provide empirical facts and relations; evidence-based policies are likely more resilient to political pressure and it is a way through which publicly funded research can make a contribution to society. Given these arguments, policy makers, regulators and standard setters, including accounting, auditing and financial regulators, are increasingly under pressure to undertake this approach. However, despite its promise, evidence-based policy making presents several challenges both in terms of how to produce quality research that can support and inform policy making, but also in terms of how research is communicated to, and used by, policy makers.
Before I discuss these challenges, let me briefly reflect upon the contributions of accounting and financial markets research to policy making. Surely, normative considerations have a long tradition in accounting, and there is plenty of potential for our research to be policy-relevant (think about the effects of accounting standards and disclosure requirements on capital markets). Yet, we do not have a systematic account of our contributions to policy making, aside from a few impact studies. These studies show that generally research in social sciences (such as economics, management, finance and psychology) has less impact than research in the natural sciences (e.g. Burton et al., 2017). However, we can still ask ourselves: what have we learnt from past research that is helpful to regulators and policy makers? Have we established facts and relations that are sufficiently reliable for policy making?
Literature surveys in accounting provide an overview and critique of the literature, but they typically do not say much about whether an academic consensus has emerged for specific policy or regulatory issues. We have some evidence that suggests that an academic consensus can be reached when the underlying body of research is large and the literature deep. We do not have evidence specific to accounting, but I would say that in our discipline there are certain economic relations that are policy relevant and for which we have enough evidence so that a consensus should have emerged. Take as an example the positive relation between corporate disclosure and market liquidity or the trade-off between relevance and reliability of financial information. Thus, while we should have insights to offer to evidence-based policy-making, we currently lack a systematic way to aggregate, synthesize and communicate the research findings on particular policy issues.
On a more positive note, I believe there are several opportunities for accounting research to enhance its external impact. Accounting and disclosure research, for example, is relevant outside its core domain: think about comply-or-explain policies in relation to corporate governance; consumer protection (e.g. product or hygiene labels); environmental disclosure (carbon emissions, mine safety) and health care (e.g. price transparency). Accounting research can also make important contributions with respect to the “real” effects of transparency mandates. By “real” effects I mean the behavioral changes by those asked to produce the information. These real effects are often close to the intent of a regulation (i.e., to incentivize desirable, or discourage undesirable, behavior).
Despite all the promise and opportunities, we would be naïve if we did not acknowledge that research supporting evidence-based policy making faces a number of major challenges. From a high-level perspective, there is debate as to whether public policy (and financial regulation) based on evidence is even feasible, given its inherent political and ideological nature. However, even putting this philosophical debate on the side, we still need to be aware of several other challenges.
First, causal inference plays a critical role for evidence-based policy making but it is also very hard to obtain. Policymaker are interested in the magnitude of the effects (for conducting a cost-benefit analysis) and the estimated magnitude really matter only when the estimates are causally identified. See Leuz and Wysocki (2016) for more details.
Second, the measurement of the regulatory “treatment” in accounting and financial markets studies is generally less straightforward than in, say, medicine (e.g., 50 mg of an active ingredient). The equivalent in economics is computing elasticities, but in standard setting and financial regulation such elasticities are very hard to obtain, given the challenges in measuring the amount and quality of information in financial reports and disclosure documents. We typically only know that a new standard or regulation was introduced, but not by how much it changed available information. In this respect, structural estimation approaches may be a potential venue to move the literature forward.
Third, probably the greatest challenge is the lack of data, which ultimately is also at the heart of causal inference and measurement problems. We lack sufficiently granular data to isolate changes in the accounting numbers that are due to the change in an accounting rule. Typically, instead, we observe changes in accounting numbers that are highly aggregated and are not solely due to the change in the rule.
The fourth challenge is related to the reliability of our research findings, which is always an important dimension to consider, but it becomes quite critical when findings are used to inform policies. Discretion in empirical analyses is at the heart of this issue, and possibly exacerbated by incentives in the publication process, such as the fact that it is generally hard to publish null results. It is possible to mitigate these concerns by pre-registering studies, sharing data and codes and implement post-publication reviews.
Finally, we need to acknowledge that policymaking is inherently political and therefore research supporting policy decisions will be subject to political influences. Research funding, access to proprietary data and any advisory relations between researchers and policy making institutions, whatever their sort, can create conflict of interest for researchers. Hence, we need proper safeguarding for the process. However, we also need be aware that we cannot just produce research and then hope that it finds its way to the policymakers. The transmission of findings is a key element if evidence is to be used systematically in policy making. In this regard, we need to be aware of the potential for regulators to cherry-pick findings in order to legitimate their choices.
Looking forward: what can we do?
My key message is that, if the goal is evidence-based, or even just evidence-informed policy making, then we need to create an infrastructure to support such an effort. This path will require a systematic and concerted effort by many researchers, research institutions and policymakers, including changes in:
1. How we organize research, for example by overcoming silos approaches by methods or discipline and building knowledge around a topic or question rather than around fields and methods.
2. How we aggregate findings and evidence, that is, establish a “canon” of policy-relevant research findings in accounting that are well understood and reliably estimated. We can do so by starting surveys in accounting similar to the Initiative on the Global Markets Economic Experts Panel, establishing “research clearinghouses” across policy areas, and developing systematic reviews of the accounting literature following the examples of the Cochrane or Campbell reviews.
3. How regulators enable research, especially to overcome the lack of data which is at the core of conducting relevant and reliable research for policy making. In particular, regulators could make their data available, require (a random sample of) firms to keep specific data around regulatory changes or involve researchers in carefully designed or randomized pilot studies.
We are still a long way from evidence-based policy making for accounting standards or financial regulation and we therefore need to temper our expectations. The aim should probably be a more modest evidence-informed policy making. In standard setting and financial regulation, professional judgement will continue to play an important role but it is worth trying to use evidence more systematically, considering the potential costs of poorly designed or implemented regulation.
Thus, in my mind, the glass is half full, but substantial investments into the research infrastructure are necessary, as is illustrated by the example of evidence-based medicine, which was also a major and decade-long effort. The risk is otherwise that we simply pay lip service to the idea and not achieve its goals.
