Re: Peer Review Reform Hypothesis-Testing

From: Stevan Harnad <harnad_at_coglit.ecs.soton.ac.uk>
Date: Mon, 13 Mar 2000 09:27:58 +0000

On Sun, 12 Mar 2000, Andrew Kenneth Fletcher wrote:

> I have a real problem with the current Peer Review System. It is biased
> towards in-house publications and outsiders are ignored.
>
> I had an idea to set up a new newsgroup titled "Peer Review Sci" I am
> certain that it would attract many professional contributors, who would
> normally have been ignored by publishers and therefore provide independent
> researchers with an unbiased review of their work.
> It would also be a far better way to make sure that nothing false arrives in
> print, because it would be an open peer review system and anyone
> contributing either a paper or a review of a paper would be open to comment
> from other reviewers. This would generate a tremendous amount of new science
> and encourage the people with the ideas to come forward.
>
> What say ye to this?

The notion of replacing peer review by some form of open commentary has
been proposed many times, in this Forum and elsewhere (and it is being
experimented with by several sites on the Web). See the other threads on
this in this Forum (1999, 1999, & 2000) and:

    Harnad, S. (1998) The invisible hand of peer review. Nature
    [online] (c. 5 Nov. 1998)
    http://helix.nature.com/webmatters/invisible/invisible.html

    Longer version below to appear in Exploit Interactive
    <http://www.exploit-lib.org/>:
    http://www.ecs.soton.ac.uk/~harnad/nature2.html

Here are some relevant excerpts from the above:

        Self Policing?

        Alternatives have of course been proposed, but to propose is
        not to demonstrate viability. Most proposals have envisioned
        weakening the constraints of classical peer review in some way
        or other. the most radical way being to do away with it
        altogether: Let authors police themselves; let every submission
        be published, and let the reader decide what is to be taken
        seriously. This would amount to discarding the current
        hierarchical filter -- both its active influence, in directing
        revision, and its ranking of quality and reliability to guide
        the reader trying to navigate the ever-swelling literature
        (Hitchcock et al. 2000).

        There is a way to test our intuitions about the merits of this
        sort of proposal a priori, using a specialist domain that is
        somewhat more urgent and immediate than abstract "learned
        inquiry"; then if we are not prepared to generalise this
        intuitive test's verdict to scholarly/scientific research in
        general, we really need to ask ourselves how seriously we take
        the acquisition of knowledge: If someone near and dear to you
        were ill with a serious but potentially treatable disease,
        would you prefer to have them treated on the basis of the
        refereed medical literature or on the basis of an unfiltered
        free-for-all where the distinction between reliable expertise
        and ignorance, incompetence or charlatanism is left entirely to
        the reader, on a paper by paper basis?

        A variant on this scenario is currently being tested by the
        British Medical Journal
        <http://www.bmj.com/cgi/shtml/misc/peer/index.shtml>, but
        instead of entrusting entirely to the reader the quality
        control function performed by the referee in classical peer
        review, this variant, taking a cue from some of the
        developments and goings-on on both the Internet and Network TV
        chat-shows, plans to publicly post submitted papers unrefereed
        on the Web and to invite any reader to submit a commentary;
        these commentaries will then be used in lieu of referee reports
        as a basis for deciding on formal publication.


        Expert Opinion or Opinion Poll?


        Is this peer review? Well, it is not clear whether the
        self-appointed commentators will be qualified specialists (or
        how that is to be ascertained). The expert population in any
        given speciality is a scarce resource, already overharvested by
        classical peer review, so one wonders who would have the time
        or inclination to add journeyman commentary services to this
        load on their own initiative, particularly once it is no longer
        a rare novelty, and the entire raw, unpoliced literature is
        routinely appearing in this form first. Are those who have
        nothing more pressing to do with their time than this really
        the ones we want to trust to perform such a critical QC/C
        function for us all?

        And is the remedy for the possibility of bias or incompetence
        in referee-selection on the part of editors really to throw
        selectivity to the winds, and let referees pick themselves?
        Considering all that hangs on being published in refereed
        journals, it does not take much imagination to think of ways
        authors could manipulate such a public-polling system to their
        own advantage, human nature being what it is.


        Peer Commentary vs. Peer Review


        And is peer commentary (even if we can settle the vexed "peer"
        question) really peer review? Will I say publicly about someone
        who might be refereeing my next grant application or tenure
        review what I really think are the flaws of his latest raw
        manuscript? (Should we then be publishing our names alongside
        our votes in civic elections too, without fear or favour?) Will
        I put into a public commentary -- alongside who knows how many
        other such commentaries, to be put to who knows what use by who
        knows whom -- the time and effort that I would put into a
        referee report for an editor I know to be turning specifically
        to me and a few other specialists for our expertise on a
        specific paper?

        If there is anyone on this planet who is in a position to
        attest to the functional difference between peer review and
        peer commentary (Harnad 1982, 1984), it is surely the author of
        the present article, who has been umpiring a peer-reviewed
        paper journal of Open Peer Commentary (Behavioral and Brain
        Sciences [BBS] <http://www.princeton.edu/~harnad/bbs.html>,
        published by Cambridge University Press) for over 2 decades
        (Harnad 1979), as well as a brave new online-only journal of
        Open Peer Commentary, likewise peer-reviewed (Psycoloquy,
        sponsored by the American Psychological Association,
        <http://www.princeton.edu/~harnad/psyc.html>), which entered
        its second decade with the millennium.

        Both journals are rigorously refereed; only those papers that
        have successfully passed through the peer review filter go on
        to run the gauntlet of open peer commentary, an extremely
        powerful and important SUPPLEMENT to peer review, but certainly
        no SUBSTITUTE for it. Indeed, no one but the editor sees [or
        should have to see] the population of raw, unrefereed
        submissions, consisting of some manuscripts that are eventually
        destined to be revised and accepted after peer review, but also
        (with a journal like BBS, having a 75% rejection rate) many
        manuscripts not destined to appear in that particular journal
        at all. Referee reports, some written for my eyes only, all
        written for at most the author and fellow referees, are nothing
        like public commentaries for the eyes of the entire learned
        community, and vice versa. Nor do 75% of the submissions
        justify soliciting public commentary, or at least not
        commentary at the BBS level of the hierarchy.

        It has been suggested that in fields such as Physics, where the
        rejection rate is lower (perhaps in part because the authors
        are more disciplined and realistic in their initial choice of
        target journal, rather than trying their luck from the top
        down), the difference between the unrefereed preprint
        literature and the refereed reprint literature may not be that
        great; hence one is fairly safe using the unrefereed drafts,
        and perhaps the refereeing could be jettisoned altogether.


--------------------------------------------------------------------
Stevan Harnad harnad_at_cogsci.soton.ac.uk
Professor of Cognitive Science harnad_at_princeton.edu
Department of Electronics and phone: +44 23-80 592-582
             Computer Science fax: +44 23-80 592-865
University of Southampton http://www.ecs.soton.ac.uk/~harnad/
Highfield, Southampton http://www.princeton.edu/~harnad/
SO17 1BJ UNITED KINGDOM

NOTE: A complete archive of this ongoing discussion of providing free
access to the refereed journal literature is available at the American
Scientist September Forum (98 & 99 & 00):

    http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html

You may join the list at the site above.

Discussion can be posted to:

    american-scientist-open-access-forum_at_amsci.org
Received on Mon Jan 24 2000 - 19:17:43 GMT

This archive was generated by hypermail 2.3.0 : Fri Dec 10 2010 - 19:45:43 GMT