by Edith Leung and Jochen Pierk

December 2022

One of the most visible activities of the EAA is the Annual Congress, which provides a forum for academics to meet and discuss research. Although the Congress serves many important purposes, such as symposia and networking, a key reason for many academics to attend is to present and receive feedback on ongoing research in one of the scientific sessions. The aim of these sessions is to help authors improve their work-in-progress for eventual publication in an academic journal, yet little is known about eventual publication outcomes following presentation at the Congress.

In this blog, we share the results of our analyses of publication outcomes for papers presented at the EAA Annual Congress and identify potential areas of improvement for the scientific presentation sessions at the Congress. We believe these insights are interesting for the academic community at large and provide a useful basis for further discussion about the merits of presenting at the Congress. These data were graciously provided to us by the EAA, but any opinions and views expressed in this article are solely those of the authors and do not represent the official view of the EAA.

In the remainder of this paper, we first outline how the EAA sessions and submission process are currently organized. Next, we provide statistics on publication outcomes for differently rated papers and papers in types of sessions. We conclude with several recommendations to further improve the scientific standard at the Congress and give more room to a more diverse set of academics at different stages in their career to attend the Congress.

Current Setting and Challenges

The Annual Congress organizes two types of sessions in which academics can present their work: the parallel sessions and research fora. In the current setting, scholars are invited to submit their research in the form of a full paper to the EAA Annual Congress, indicating their method of choice and the research area of the paper. Subsequently, the Standing Scientific Committee (SSC) members responsible for a particular research track allocate two reviewers to each submission. Reviewers assign scores between 1 and 6 and are encouraged to provide written feedback that may be useful to the authors of a paper. Based on these reviews and scores based on the evaluation criteria, the SSC of the EAA then decides upon the acceptance or rejection of papers. Usually, papers that receive a score between 2.5 and 4 are accepted for presentation at the Annual Congress in a Research Forum session (RF, 90 minutes for 5 papers), while papers with a score above 4 enter a Parallel Session (PS, 90 minutes for 3 papers) or Parallel Session with Discussant (PSD, 90 minutes for 2 papers, including an assigned discussant).[1]

The implicit assumption of the evaluation criteria is that Research Fora allow for the discussion of viable and interesting research projects at an earlier stage, that need further development to be reasonably submitted to a journal. For example, a score of 2 is described as ‘initial work on a potentially viable project, but it is not likely to be ready for submission for some time’. However, based on our anecdotal impressions, the public perception of RF sessions seems to be that they often include low-quality, rather than early-stage, papers. A potential reason is that reviewers are not able to distinguish between an early-stage paper and a low-quality paper (both would be assigned a low grade). As such, one goal of our analyses is to determine whether this perception is supported empirically by worse publication outcomes of papers presented in the RF, and we give recommendations to improve the quality of presentation sessions at the EAA.

Publication Outcomes of Presented Papers

Sample

We start by gathering papers published in the top 15 accounting journals (based on 2020 impact factors) and Accounting in Europe.[2] In total, these 16 journals published 2,366 papers between 2012 and 2021. While this selection does not include all potential target journals, especially topical journals are missing, we believe that it covers at least a relevant set of journals for most scholars, including the journals of the European Accounting Association. 

We gather the acknowledgments of all these papers from Scopus and scrape the missing acknowledgments from the journal websites. 608 of these publications mention in the acknowledgments that the paper was presented at an EAA conference. We then manually match these 608 publications to the 6,718 EAA conference presentations between 2012 and 2019. In total, we successfully match 418 journal publications to EAA conference presentations. Most of the remaining 190 journal publications that mentioned the EAA in the acknowledgments were presented before 2012, and in a few incidences, we could not find a match.

Results

Table 1 shows the number of papers published in the 16 accounting journals with prior EAA presentations separately by journal. As one might expect, the European Accounting Review (EAR) includes most publications with prior EAA presentations, which highlights that authors often present their work at the conference before submitting their paper to the EAR. 

Table 1: Publications with prior EAA presentation

Note: this table documents the number of publications per journal of papers that mention presentation at the EAA Annual Congress between 2012 and 2019. See footnote 2 for explanation of the abbreviated journal names.

In Table 2, we document the association between the average reviewer score and the likelihood of publication. We find that the likelihood of publication increases with the reviewer scores, which indicates that reviewers, on average, do a good job of assessing the quality of papers. The reviewer instructions of the EAA include the threshold of an average reviewer score of 4 (called ‘anchor’ on the conference website) to indicate if a paper is ‘ready’ for submission to a journal such as EAR or AIE. While we do not see a clear-cut threshold in the likelihood of publication in Table 2, publication likelihood for papers with scores 4 and above is much higher than for lower-scoring papers. Especially papers that score below 3 are very rarely published in the selected 16 journals. It is important to note that we do not imply that these papers are necessarily of low quality or are never published (e.g., these papers may be published in journals that are not included in our selection). We also caution that these statistics should not be used to determine if a paper has ‘potential’ or not, since many factors determine publication outcomes and reviewer scores are to some extent subjective.   

Nevertheless, only one paper out of the 883 presented papers with a score below 3 was published in the association’s journals EAR and AIE. It is reasonable to assume that the paper quality was often not sufficient for high-quality journals, in line with the conference reviewers’ assessment that these papers are not ready for submission. In unreported analyses, we also do not find that these papers are published with delay. Taken together, these statistics indicate that the quality of low-scoring papers is, on average, too low to warrant publication in these journals.

Table 2: Publications by Reviewer Score

Note: this table presents the count of papers by reviewer score that were presented at the EAA Annual Congress between 2012 and 2019, and the number (percentage) of papers by reviewer score that are eventually published in the journals listed in footnote 3.

Recommendations

Better distinction between early-stage and developed papers

Our analyses suggest that publication likelihood increases with reviewer scores, suggesting that reviewers do a good job of assessing the quality of papers. At the same time, the evaluation criteria and score descriptions provided to reviewers indicate that lower scores should primarily reflect the development stage of the paper. However, low-scoring papers are rarely published in academic journals, rather than published at an equal rate but with a longer delay. Hence, it highlights that the current review system does not allow reviewers to differentiate the development stage of a paper from its underlying quality. To more effectively do so, we suggest adjusting the scoring and acceptance criteria and submission system to better distinguish early-stage and more developed papers as follows:

• Allow authors and ask reviewers to indicate whether a paper is early-stage work or a developed paper (the latter to corroborate authors’ assessment).[3] Early-stage work would be more in the form of proposals (e.g., a short document with a page limit that explains the what-why-how of a paper, with some preliminary results). Full paper submissions should be papers that could reasonably be submitted to a journal such as EAR (including all relevant sections of the paper).
• Conditional on whether a paper is early-stage or fully developed, reviewers can then assess the likelihood of publication in a good outlet such as EAR as in the previous scoring system. The simple differentiation between early-stage and developed papers should enhance the SSC’s ability to compile a program that only accepts papers that are of sufficient quality, independent of its current development stage. The exact thresholds that determine inclusion in a presentation session should ensure that access to the EAA Annual Congress remains fair and achievable for a diverse set of academics.
• In addition, the explicit allowance for early-stage work may also encourage authors to submit work that fits these criteria (i.e., increasing the number of submissions), which could offset the higher rate of rejection in fully developed but lower-quality papers.
• Furthermore, early-stage projects benefit from timely feedback, which could be especially interesting to Ph.D. students and researchers at the beginning of their careers.

Parallel sessions vs. research fora

Although the research fora are not intended to be “lower quality” sessions but an outlet for discussing earlier-stage work, anecdotal evidence suggests that some academics view research fora as inferior to parallel sessions. To level the playing field, some suggestions below:

• To combat the negative connotation with research fora that some authors and academics have, we would recommend replacing research fora with so-called “early-stage parallel sessions”. Early-stage parallel sessions would still include 4-5 papers, similar to the number in the current research fora, but together with the previous recommendation of revising the submission system, we can signal to our academic community that there is a real change and improvement in the sessions at EAA.
• Moreover, we recommend introducing discussants for early-stage parallel sessions, since early-stage work, in particular, would benefit from elaborate feedback from a well-prepared discussant. Next to the opportunity for high-quality feedback from a discussant, the introduction of discussants at early-stage presentations is valuable for the network of presenters at such sessions, since these presenters are potentially more likely to be in earlier stages of their career (e.g., PhD students).
• In short, the proposed set-up would be to have 90-minute sessions divided as follows:
  • Full paper session with discussant (PSD) including 2 papers per session.
  • Full paper session (PS) including 3 papers per session.
  • Early-stage parallel session with discussant (ESPSD) including 3 papers per session.
  • Early-stage parallel session (ESPS) including 4 to 5 papers per session.

We recognize these suggestions are subject to organizational feasibility (e.g., room constraints at the hosting venue, or ability to organize more sessions to accommodate the ESPSD proposal). However, we believe that the first step of revising the submission process to screen for paper quality more effectively, together with the introduction of ESPS(D), should result in a similar number of submissions and presentations, reducing the practical challenges of implementing these recommendations. Moreover, we also hope these suggestions will also help increase access to the EAA Annual Congress for a more diverse set of academics.

[1] See: https://eaa-online.org/congress-2022/submission-review-process/

[2] These journals are: Accounting and Business Research (ABR), Accounting, Auditing, and Accountability Journal (AAAJ), Accounting and Finance (A&F), Accounting Horizon (AH), Accounting in Europe (AIE), Accounting Organization and Society (AOS), British Accounting Review (BAR), Contemporary Accounting Research (CAR), European Accounting Review (EAR), International Journal of Accounting Information Systems (IS), Journal of Business Finance & Accounting (JBFA), Journal of Accounting and Economics (JAE), Journal of Accounting and Public Policy (JAPP), Journal of Accounting Research (JAR), Review of Accounting Studies (RAST), and The Accounting Review (TAR).

[3] One potential problem with the suggested set-up is that authors might try to ‘game’ the system and submit their fully developed paper to the early-stage sessions to increase the likelihood of acceptance. However, we believe that this is not problematic as reviewers will likely also assign a low score for the early-stage session if the paper lacks contribution. Furthermore, even in the current set-up, these papers would likely be accepted for the research forum.