The Bled Conference Guidelines for Paper Reviewers

Author: Roger Clarke
Prepared for the 19th Bled eCommerce Conference 5-7 June 2006, when presenting the Outstanding Paper Award
© Xamax Consultancy Pty Ltd, 2006

Available under an AEShareNet  licence or a Creative Commons  licence.

Version of 31 July 2007 (available at: http://www.rogerclarke.com/EC/BledRevGuide.html)

Introduction

The quality of reviews is very important to any conference community. The Bled Conference benefits from the efforts of c. 200 people who review papers each year. Although the conference is fortunate in having such a large community of committed scholars, and the quality of reviews is generally high, it is important that the quality be sustained and where necessary improved.

So reviewers are requested to ensure that their reviews reflect the guidance provided in this document.

The Function of Reviewing

The purpose of reviewing a paper is to assure quality, in both the negative sense of filtering out low-quality papers, and the positive sense of encouraging the Author(s) to deliver a better version of the paper in question, and better papers in the future.

What represents ‘quality’ differs between journals and conferences, and between conferences. The Bled community is special, because it comprises both academics and practitioners. The community wants contributions that make a difference, and whose significance they can grasp.

In short, the Bled Community is interested in papers that satisfy the following all-too-rare quality factors (bad wordplay intended):

R elevance
A mbition
R igour
E ase of Access

Who is the Reviewer Working For?

In brief, the Program Chair, the paper’s Author(s), and the Bled Conference Community.

The foreground purpose is to provide the Program Chair with information about the paper’s qualities, and its acceptability for the conference. The background purpose is to inform the Author(s) what needs to be done to improve the paper. All members of the Bled Community stand to gain from a professional approach to reviewing, because it enhances the quality of the papers that get accepted into the Program.

Indicators of Quality in a Review

Here are the key characteristics of a review that meets the needs of the Program Chair, the Author(s) and the Bled Conference Community alike:

1. Demonstrated Understanding of the Paper. Briefly summarise the paper, or convey by other means that you’ve grasped what the author was trying to do. If the readers of a review aren’t satisfied that the reviewer understood the paper, they will de-value the comments

2. Positive Features of the Paper. This should be brief, but it should be present. Criticism is more likely to be effective if it is cushioned by some recognition of the author’s worth. It’s unusual for a paper to reach a reviewer without at least some redeeming features

3. Critique, Expressed Constructively. This is the central feature: a good review must lead somewhere. The Program Chair needs to make a decision; and the Author needs suggestions about what they can do about the features the reviewer doesn’t like. If your opinion is that the paper isn’t appropriate for Bled, is there some other venue where it might find a home? Or are there steps that the author could take to improve their chances next time they submit a paper?

4. Value-Add. Performing a review is an act of collaboration, not competition. In addition to identifying and helping to eradicate inadequacies, offer additional perspectives and references

What Not To Do

Here are some traps that reviewers need to avoid. The first few are corollaries of the four quality indicators discussed above:

• Don’t be vague about the paper’s weaknesses. It helps neither the Program Chair nor the Author to say that ‘this paper makes too limited a contribution to be considered for this venue’, ‘the method is weak’, ‘the literature search is too narrow’, or ‘the implications are insufficiently developed’. Comments need to be specific, and supported by evidence

• Don’t be emotive, aggressive, sarcastic or demeaning. This shakes the confidence of junior or otherwise insecure authors, and in any case reduces the effectiveness of the advice provided

• Don’t criticise without ‘actionable advice’. This term is Allen Lee’s. He argues that a criticism shouldn’t even be made unless it includes a suggestion about what might be done about it

Other issues are ethical in nature:

• Don’t delegate unless you exercise control over the quality of the review. It is unfair to the Program Chair and the Author(s) to assign the task to a junior colleague or a student, unless you review their draft comments and the paper itself, adapt the draft as necessary, and take personal responsibility for the review that is submitted in your name

• Don’t fail to declare a conflict of interest. Examples include an association with the research reported on, or with one of the authors of the paper. (This need not disqualify you from doing the review; but the relationship needs to be known)

• Don’t fail to declare the limits of your expertise. A partial lack of expertise (e.g. limited knowledge of the research domain or the specific sub-area or the particular research method) does not disqualify you from doing the review (otherwise some papers would never find a reviewer!). But the scope of the problem needs to be declared to the Program Chair and Author(s) alike

• Don’t take premature advantage of content. Prior to publication, don’t provide copies to a colleague or student who works in the same area (unless you approach the Program Chair and get the Author’s consent to early distribution)

Some Examples of Good and Bad Content in Reviews

Bad

  • The paper is not up to the standard expected for this conference. (Vague, not actionable)
  • The paper needs to be structured better. (Vague, not actionable)
  • There is room for improvement in the literature review and theory building areas. (Vague, not actionable)
  • Problem definition, argumentation and conclusions are far away from what is expected. (The authors clearly need guidance, but they haven’t been given any)
  • A number of language mistakes and incoherent sentences could be solved. (Which??)
  • Some parts of the paper could be rewritten in such a way that they improve comprehension to the less-initiated. (Which??)
  • Some paragraphs had to be read more than once, in order to be fully comprehensible. (Which??)

Good

  • The author/s need to spell out briefly the aims and objectives of the research at an early stage (ideally in the introduction) in order to set a framework for the rest of the paper.
  • [In Figure 1,] the semantics of the boxes, circles, and lines should be more precisely articulated.
  • The big problem of the paper is that the authors failed to clearly define the factors listed [in Figure 1].
  • The paper is a mix of a research project description & a discussion of empirical findings & thus lacks focus. I would be interested in both aspects but since neither part provides an in-depth discussion, I found the result not very satisfying.
  • The paper’s weaknesses are the lack of in-depth case analysis and the lack of a strong theoretical discussion. My suggestion is to make more explicit what the theoretical focus of the article is and build the story line around that focus.
  • We learn very little about the set-up of the study. Who were the participants? How were they selected? What sites did they visit? How often? When? What were the demographics, and did participants have differing experience and skills with respect to online buying? Can the study still be representative, given the low response rate? Was a non-response-bias identified? And in what ways (if any) can the results be generalised outside the geographical area in which the study was performed?

Bibliography

Directly relevant articles and notes from within the IS discipline include:

Bieber M. (1997) ‘How to Review’

Davison R.M., de Vreede G-J. & Briggs R.O. (2006) ‘On Peer Review Standards For The Information Systems Literature’ Commun. AIS 16, 49 (2005) 967-980

Drawing on the literature, Davison et al. (2006, p. 15) identifies attributes of a good reviewer as being humaneness, competence, openness, unbiassedness and absence of prejudice, ethicality, timeliness, persuasiveness and diligence

Koh C. (2003) ‘IS journal review process: a survey on IS research practices and journal review issues’ Infor. & Mngt 40 (2003) 743–756

Lee A.S. (1995) ‘Reviewing a Manuscript for Publication’ Invited Note in J. Ops Mngt 13, 1 (July 1995) 87-92

Zmud R. (1998) ‘A Personal Perspective on the State of Journal Refereeing’ MIS Qtly 22, 2 (September 1998)

A couple of relevant publications from other disciplines are:

Black N., van Rooyen S., Godlee F., Smith R. & Evans S. (1998) ‘What Makes a Good Reviewer and a Good Review for a General Medical Journal’ J. Am. Med. Assoc. 280, 3 (1998) 231-233

Finney D.J. (1997) ‘The Responsible Referee’ Biometrics 53, 2 (June 1997) 715-719

Hames I. (2007) ‘Peer Review and Manuscript Management in Scientific Journals’ Blackwell/ALPSP, 2007, from Blackwells

Smith, A.J. (1990)  ‘The task of the referee’ IEEE Computer 23, 4 (April 1990) 65-71