Guidelines for reviewing

Lab discussion on reviewing

The material on this website was collated by Marjorie, Bill and Caroline for a mathematical ecology lab discussion on March 8th 2006.

presentation file prepared by Marjorie Wonham.

Peer review papers

Papers about the review process including rejection

Siegelman, S. S. (1991). “Assassins and Zealots – Variations in Peer-Review – Special Report.” Radiology 178(3): 637-642. (pdf)

Study of 660 referees who rated more than 10 papers over 4.5 year period. The average rating was 4.8 (4 = borderline, 5 = uncertain). A normal distribution of reviewers ratings was found with 87.4% of reviewers in the mainstream. Zealots, pushovers, demoters and assassins are also characterised. Assassins and demoters claim they are trying to improve the standard of accepted papers. Zealots and pushovers would like to have more papers appear within their areas of expertise, often providing detailed critiques useful for revisions. Journals that monitor reviewers are more likely to obtain a fair overall judgment of a paper.

Cassey, P. and T. M. Blackburn (2003). “Publication rejection among ecologists.” Trends in Ecology & Evolution 18(8): 375-376. (pdf)

A survey of ecologists. Everyone experiences rejection. Most authors will have each paper rejected by at least one journal. Respondents thought publishing was getting harder and unfair if they had experienced rejection recently, otherwise, thought it about the same as 10 years ago and peer review a fair process. 72% of respondents have at least one manuscript they have failed to publish anywhere. Rejection is not a handicap in career advancement. Ecologists need a thick skin. If at first you don’t succeed, try again.

Bordage, G. (2001). “Reasons reviewers reject and accept manuscripts: The strengths and weaknesses in medical education reports.” Academic Medicine 76(9): 889-896. (pdf)

Top 10 reasons for manuscript rejection: inappropriate or incomplete statistics; overinterpretation of results; inappropriate or suboptimal instrumentation; sample too small or biased; text difficult to follow; insufficient problem statement; inaccurate or inconsistent data reported; incomplete, inaccurate, or outdated review of the literature; insufficient data presented; and defective tables or figures. Main strengths of papers accepted: timeliness of problem studied and contribution to the field; excellence of writing; and soundness of study design. The level of agreement among reviewers is highly variable. 33% of reviews were unanimous (18% to accept, 15% to reject). Written comments are required in addition to numerical scores for an editor to be able to decide on a manuscript. 6 recommendation to researchers and authors: pay attention to relevance (theoretical or practical), select optimal study designs, select optimal instruments, select optimal statistics, interpret the results honestly, and present well-written manuscripts. Acknowledging the limitations of a study are very important.

The responsibilities of reviewers and ethics of reviewing

Wilson, J. R. (2002). “Responsible authorship and peer review.” Science and Engineering Ethics 8(2): 155-174. (pdf)

Science is based on the building blocks of ascertained facts found by their predecessors. Scientists should provide honest records of their work. Do unto others as you would have them do unto you.” It is an ethic that should govern everyone: authors, editors, or manuscript referees. Principles of authorship are covered. The main purpose of the peer review system is provide expert advice (for publications and grant proposals). Apart from the acceptance or rejection decision, reviewers can improve the quality of a manuscript through thoughtful suggestions and feedback. However, peer review cannot absolutely detect fraud or certify the validity of manuscript. Problems arise in the peer review process if editors don’t take responsibility for difficult decisions, or reviewers who don’t read or evaluate the work. Ground-breaking studies can be rejected because of the lack of imagination of a reviewer. A referee should answer 3 questions: Is it true? Is it new? Is it interesting? Table 2 contains key questions to be answered by a referee’s report. Reviewing can help improve the quality of science. Reviewers should take pleasure in intellectual stimulation, interaction and education that arises from reviewing.

Benos, D. J., J. Fabres, et al. (2005). “Ethics and scientific publication.” Advances in Physiology Education 29(2): 59-74. (pdf)

Discusses ethics in science including fabrication, falsification, plagiarism, redundant and duplicate information, conflict of interest, authorship, welfare of animals and humans, and reviewer responsibilities. The Office of Research Integrity and mechanics of scientific misconduct complaints are covered. Misconduct cases represents a small fraction of all manuscripts in the system. Case studies and discussion questions for each type of ethical issue are provided. Being an author is a privilege and should signify a personal contribution and impart respect and pride. Information on author guidelines and responsibility. Reviewing responsibilities covered. It is an honour to be selected as an expert to review a manuscript. Essential to ensure that only the best and solid science appears. Submitted manuscript should not be retained or copied by the reviewers. Reviewers and editors should not make any use of the data, arguments, or interpretations, unless they have the authors’ permission before publication. Reviewers should provide speedy, accurate, courteous, unbiased, and justifiable reports. A referees report should consider the following:

  1. evaluating the manuscript honestly, objectively, and critically;
  2. disclosing (or avoiding) any real or perceived conflicts of interest with the work or the authors;
  3. not engaging in plagiarism;
  4. identifying to the editor areas of the manuscript in which the reviewer is not an expert;
  5. writing reviews in a constructive, helpful fashion, and not being derogatory;
  6. reviewing expeditiously;
  7. maintaining confidentiality;
  8. reporting any suspected ethical breach to the handling editor.

The mechanics of writing a review

Benos, D. J., K. L. Kirk, et al. (2003). “How to review a paper.” Advances in Physiology Education 27(2): 47-52. (pdf)

Most scientists acquire review training by actually doing it. Repeated bad reviews likely result in not being asked again. Proper training should be given through guidelines and responsibilities. This is essential for making the review process valuable. As a journal advocate, the reviewer’s job is to make sure that the best possible science appears in print. The purpose of peer review is to ensure

  1. quality, checking that no mistakes in procedure or logic have been made;
  2. that the results presented support the conclusion drawn;
  3. that no errors in citations to previous work have been made;
  4. that all human and animal protocols conducted follow proper review and approval by appropriate institutional review committees;
  5. that the work is original and significant.

The final decision lies with an editor, but reviewers should give a substantive evaluation of a paper and back up their statements. Reviewers should only comment on subject matter that is within their area of expertise. Treat all manuscripts in the manner you would want yours to be treated. Table 1 contains criteria for manuscript review. Ethics of reviewing are also covered, as per Benos et al. 2005. It takes time to write a useful and critical review and is the responsibility of a referee.

Roberts, L. W., J. Coverdale, et al. (2004). “How to Review a Manuscript: A “Down-to-Earth” Approach.” Academic Psychiatry 28(2): 81-87. (pdf)

This paper is concerned with the elements of peer review. Accepting to do a review should not be delayed. The review should be submitted in a timely manner. Reviews should comment on the strengths and weaknesses of a paper, but reviewers comments are often negative which is discouraging for authors. Try not to be unduly harsh but constructive instead. A very helpful section by section approach is given which covers how entire review could be carried out. A checklist (pp.86-87) is included for reviewers to work through while completing a review.

Provenzale, J. M. and R. J. Stanley (2005). “A systematic guide to reviewing a manuscript.” American Journal of Roentgenology 185(4): 848-854. (pdf)

This paper is another step-by-step guide to writing a review. It contains a checklist for completing a review and guidelines for referees (Appendix 1). Tips for giving the editor a most informative review, in terms of a global rating summary are given.

Guidelines for referees and an example review taken from the International Forestry Review. (pdf)

This document has some helpful hints, and also wording for constructive criticism that might be useful.

  • Checklist for reviewing a paper in mathematical biology. (pdf)

These are guidelines written by David Gavaghan for a lab meeting in Oxford.

  • Proof corrections marks list. (pdf)