This post is based on an “interview” I did for publons a couple of years back. I hope that you find it interesting and useful.
– At what stage does an early career scientist become qualified to peer review?
Once they have graduated with their PhD and/or when they have published one or two of their own relevant papers in reasonable quality journals (I think that its important to have had firsthand experience of the “process”, especially for the journal in question). Ideally, in the first few reviews, a mentoring process can/should take place in which the early career scientist can seek feedback and guidance from a trusted senior colleague (e.g. ex-PhD supervisor). This mentoring process will be especially important in terms of “calibration” – getting the level of critique about right for the quality of the journal.
– Any tips for how these scientists might get noticed by editors for the first time?
Editors are after serious, knowledgeable and reliable reviewers. Editors, have their pool of senior people to which they will always be on the lookout for new up and coming researchers. There is nothing special to get “noticed for the first time”. A few suggestions:
– When should a reviewer reject a review invitation?
Reviewing is an important “community service” which we should all factor in as a highly worthwhile and important task – active researchers should devote meaningful effort/time to reviewing duties over the course of the year. Its important to recognise that such reviewing demands will be “lumpy” and that your experience as an author could well differ from your responsibilities as a referee – remember its not “payback” time (eg either in being slow to review or being unreasonably harsh)! I don’t often reject review invitations, but this can happen when:
One more common scenario that I am now more likely to reject reviewing requests, is multiple reviewing requests from the same journal in quick succession. This can easily happen where there are multiple co-editors and they do not “talk” to each other. A clear example of this that I have experienced is being asked to review a second paper for a given journal, before I have had a reasonable chance (say within 4 weeks) of completing the job on the first one. This can be very frustrating and even annoying, because I don’t want to appear to be a “shirker”, especially those journals that I value and strive to publish in myself.
– Following on from the above, do you have any rules of thumb for how many review invitations you accept per month?
No, I do not have monthly quota or any other time-based limit. However, I do try and keep my total research-related activity workload manageable (reviewing being a part of this bigger picture) and, as long as there are not major urgent tasks existing, I am happy to let my reviewing list creep up to 4 or 5 “live” jobs at any one time. But, I like to keep it down below this most of the time. This is usually achieved by making sure I don’t sit on any reviews longer than about 4 weeks – sometimes I go over this a little bit, but mostly I meet that deadline. The only problem with this approach is that you can be a “victim of your own success”, because if you are a “quick-turnaround” reviewer, editors tend to like to (“over”) use you as a “go to” referee! But, it is much better to be viewed in this way, than to be on an editor’s “black list”.
(2) Writing the review
– How do you go about reviewing a manuscript?
(For part of the answer to this question, see also the answer to the next question below.)
First, I read and think with a red pen in my hand. I annotate and make notes on the paper as I go. At least half my “scribbling” is just to underline key words, to summarise key points or arguments so that I better understand as I read at the time and also to allow me to quickly “tune in” to the paper again later, when I have inevitably put it aside for several days or a week. Then there are the notes that I make on the hardcopy, beside the relevant parts of the paper, that will likely give me the essence of a comment I will write in my report. One comment that I often write on the very first page, even only after a minute of reading, is “too long!” It’s a common problem – many authors find it very difficult to keep their words under control. Moreover, such a scenario is also tied up with “too many tables/ too many figures”. Indeed, there are a common set of issues that I see over and over again and each of my reviews will build around them, the extent to which they are at play.
While I am happy to take on the merits of each paper as I read through, I do have a pretty systematic structure in mind of what I am looking for – or more particularly, what experience tells me are the likely areas of problem/concern. Specifically, I always think through the following dimensions: Paper title; Research question; Key papers; Motivation; Idea; Data; Tools; What’s New? So What? Contribution? Other Considerations? Indeed, this is a skeleton summary of the key elements I have used to design a 2-page template for “pitching research” [freely available at SSRN: https://ssrn.com/abstract=2462059] and I find it equally useful as a framework for evaluating research at the “end” – once it has been executed and written up. Of course, the big and ultimate question is: for the journal in question, does the paper make a sufficient incremental contribution to the relevant literature?
Also, I ask myself many questions like: Is the paper a good “fit” for the journal? Are there predictions/ hypotheses in the paper? If not, why not? Do the predictions come from relevant and meaningful theory? Do they make sense? Have the authors clarified the economic significance of their key statistically significant findings? Does the paper flow – does it have connectivity? Does it have clarity – have the authors been careful to apply paper “craft” in their writing?
All of the above give me the source material for writing my report, which I try to compile in a logical way grouping similar things under a common heading where possible, and numbering them for ease of reference. Indeed, I like to give each point a short reflective heading to help the authors understand the core issue. I like my reports to be about 2-3 pages, 4 maximum. As an author and an editor myself, I have seen much longer reports which I think are “overkill” and/or a sign that the reviewer has another agenda.
– How much time do you typically devote to reviewing a single manuscript?
The one BIG advantage of having done many, many journal reviews is that you get very good and highly efficient at it. It certainly is the case of much practice pays off – especially in cases where the topic of the paper is right in your “zone” [but, even better, with more experience over time your “comfort zone” actually expands]. So, now days, I guess in many such cases I can do a review in 2-3 hours all up – but never in one sitting. For example, I typically save up the initial reading of the assigned papers on plane flights – indeed, I find reading on planes can be terribly efficient. A one hour flight is usually a nice “capsule” of time to read and annotate a paper. Importantly, I “scribble” a decision on the front page – the 3 main cases would be “”major R&R”; “Reject”; “minor R&R”. Sometimes I recommend conversion to a “note” paper. Yes, of course sometimes I recommend “accept” or “conditional accept” – but these never occur on the first round! Then days later, when I feel an opportunity arrives to write my report, I sit down and quickly refresh my memory of the paper’s content. The annotations (in red pen) are crucial here – and also the decision “scribbled” on the front. Writing up the report usually takes another hour or so and then 10 minutes to log into the journal website to submit the report and answer the set questions posed by the editor. The above is the “best case” scenario – when I know the literature and the recommendation is clear in my mind. For example, if I don’t know the literature well enough, I might need to locate and read a few key papers – this can add several hours to the task. Of course, this is more likely to be the case when you first start out reviewing as a novice researcher.
(3) Open and transparent peer review
– Do you ever sign your name to your reviews?
No. The process is double blind review and I think this is important – especially, given that the vast majority of decisions being “reject” or “major R&R”. Knowledge of the referee can have very little upside, and possibly nontrivial downside – directly and indirectly.
– Have you ever published any of your reviews? If so, what did you think of open review?
No. I am reluctant to do so because I think anonymity is a very important part of the process.
(4) Miscellaneous
– Why do you review?
Reviewing is an important part of the “community service”. For viability, sustainability and growth in quality scientific outcomes, the system needs many dedicated reviewers. As active researchers, we all benefit from having our work reviewed – its only right and fair that we do our part to support the system. Also, you learn a lot from reviewing. You read papers that you probably wouldn’t otherwise read, and you learn new techniques and even “unusual” papers can help germinate your own research thinking.
– Has your reviewing behaviour changed since you’ve started getting credit for your reviews? And if so, how?
No, my approach hasn’t changed at all.
– What (if anything) needs to change in peer review?
Perhaps just coming up with fairer system to share the reviewing load.
– Any other tips for less experienced peer reviewers out there?
