On Wed, 10 Feb 1999, Alexandre Soares wrote:
> I've read the original articles mentioned in your message and also your
> article on peer review (Harnad,S.,1998. The invisible hand of peer
> review. Nature [online] (c. 5 Nov. 1998)
> http://www.ecs.soton.ac.uk/~harnad/nature2.html). If you will, I
> would like to add another possibility. Why not let the authors
> themselves do the job of editors and publishers?
Some of your points are already considered and answered in the
Nature online paper above. Here is a list of the problems with such
proposals:
(1) A large part of the function of refereed journal publication
(including the level of the journal in question, in the quality
hierarchy of journals) is to allow the otherwise overwhelmed reader to
calibrate his reading in the enormous and growing mass of raw
manuscripts. If all publication were self-publication -- even if
accompanied by referee reports (and even if the referees were the
appropriate experts) -- this would provide no calibration for the
reader. Everything would be on a par, and one would have to plow
through paper by paper, reading now not only the papers, but the
referee reports, in the hope that it was a paper one would have wanted
to read in the first place, had it been accepted by a refereed journal
of a designated quality.
(2) So the filtering and tagging function of refereeing would be lost.
The correcting and improving function of the referee reports would also
be compromised or lost, because authors could pick and choose whether
they followed the referees' advice (having already picked and chosen the
referees in the first place). I expressed some doubts about the
self-policing capacities of human nature in the above article. Ask
yourself how likely you yourself would be, under the weight of a heavy
workload like that of most of us, to voluntarily elect to do further
work on your paper at the behest of a referee, when you have the option of
"publishing" it without doing so -- with an accompanying scolding
public referee report being the only penalty? Surely there is a reason
why self-monitoring has not been adopted as the (sole) means of
quality-control in any other expert field of human endeavour either:
At the population level (we are not speaking of the occasional
individual saint, here) we must be protected from ourselves if quality
standards are to be maintained. Otherwise there are just too many
incentives to cut corners.
(3) Public commentary (whether peer or non-peer, qualified or unqualified)
is after-the-fact. It cannot serve as the advance quality filter.
(4) Why should referees, with their loads, be any happier to lend their
services to such an unpoliced, untagged system than readers would be to
lend it their eyes? Today, when a referee is called upon to sacrifice
some of his precious time to peer review, the call comes from the
Editor of a journal of some distinction who has judged that referee
suited to review that paper, and will see to it that the referee's
recommendations -- if he judges them justified -- will be followed. In
the proposed alternative, it is not a (presumably) qualified and
respected and answerable editor who calls for the referee's services
and will see to it that they are put to use, it is the author, who can
then pick and choose. Will referees lend their time to such a
self-policing system? Indeed, as referees are human too, will they then
not rather ingratiate themselves to the great and good among authors,
and leave the calls of lesser authors for review unanswered? These too
are the wages of self-policing.
(5) Last, what improvement is this over my own proposed alternative,
which is to solicit expert opinion informally to the best of one's
ability, do the recommended revisions one judges justified, and then
archive the preprint publicly on the Web, not calling it a "refereed
journal publication" (which it isn't) but an unrefereed preprint (which
it is), while at the same time submitting it for refereeing to the
appropriate peer-reviewed journal? The final, revised outcome of that
process too (assuming it is successful) could then also follow
the preprint onto the Web tagged as a refereed reprint. Volunteer peer
commentary could be invited or spontaneously provided for both the
preprint and the reprint, likewise on the Web. What would be lost in
such a system that would be gained by the proposed self-policing system
-- whose main casualty would be a peer-reviewed literature?
> For example, the author
> would write her/his paper and choose 3 referees randomly from a pool of
> expert referees (see 1).
Can a pool of qualified experts be constituted in advance for every
potential specialty and subspecialty? (If so, why have journal editors
not yet discovered and capitalized on this remarkable resource?) Are we
more expert to pick the relevant experts who will adjudge our own
expertise than qualified, respected disinterested experts, appointed by
and answerable to the learned community would be? (Will referees
respond to our call?) Are there any empirical data that confirm the
assumption that anonymity need not be an option in order to elicit
full, frank, competent reviews of the quality of a raw manuscript?
> The referees would give their reports and the
> author would make the changes which are agreed upon and the ones which
> are not agreed would appear as appendix on the paper in a section called
> "Referees comments" together with the author's answer.
Is there any evidence that this is what quality control is or ought to
be: The candidate picks and chooses what to fix, and the rest appears as
a warning label? Would referees see this as a good use of their time?
> Obviously, in
> this section would appear general comments related to ideas (hypothesis,
> methods, applicability), not structure (grammar, etc) of the paper.
> There could be also comments from the referees about the importance of
> the paper, and referee questions and authors answers (along the lines of
> some proceedings of symposia).
This sounds like peer commentary to me: Potentially useful, potentially
even more-so on the Web; but is it a substitute for peer review?
> The referees are therefore non-anonymous
> to both authors and public (their addresses SHOULD appear in the paper
> for contacts).
Any data at all that (1) referees would go for this? or that (2) the
results would be better/worse/no quality control? Or does non-anonymity
have some sort of face validity? Should our names be published on the
walls with our votes in elections too?
> The author would then make the final layout of her/his paper (using any
> of the almost infinite variety of available softwares) and would
> "publish" the final version in her/his web site.
Why not just "publish" it on the website and then go on to get it peer
reviewed too?
> In this way we achieve several highly desirable qualities :
>
> 1- authors would have the copyright of their work, (as suggested by
> Bachrach S. et al. (1998). Intellectual Property: Who Should Own
> Scientific Papers? Science 281 (5382):1459-1460. in
> http://www.sciencemag.org/cgi/content/full/281/5382/1459),
They could have that anyway, if they insisted -- and they will. Surely
quality control need not be sacrificed in order to retain ownership of
one's intellectual property. (That Bargain would be a Faustian one
indeed!)
> 2- publication would be free of costs for both authors and readers,
> rapidly "published" and easily accessible and distributed,
It would also be free of both quality control and the signposting that
went with it, to guide, respectively, the author and the reader.
> and
>
> 3- the hand of the referees would be "visible", eliminating "invisible
> friends" and "enemies".
On the assumption that visibility is an end in itself and would be at
least as effective as optional anonymity in eliciting valid feedback...
> Rejected papers
Rejected? Who would be rejecting? Referees only recommend: In your
system it is the author who (self)-accepts or rejects!
> could be re-submitted with new randomly choosen
> referees,
In other words, one could keep shopping for referees till one found
more favorable ones. (An excellent formula for getting statistical
significance too: keep calculating the cumulative F ratio and raising
the N by gathering more data till you hit a point where p happens to
peak momentarily above .05, and then promptly stop testing and publish!
Our journals would be filled with a lot more significant effects then,
assuming referees could eventually be found to endorse them...)
> but the reports of all referees SHOULD appear. Even in the
> extreme case that the paper is rejected several times, the author
> her/himself would have the final decision on "publishing" or not, as
> long as she/he would always include the veridict of the referees. The
> wide public would then judge whether to trust or not in the results of
> the paper.
If they had the time to first wade through all those papers and referee
reports. (It almost makes you want to bring back Editors to do that
for you...)
> Notes:
>
> (1) This pool of referees could be organized by the appropriate
> scientific society and could be available in the web in a especial
> software.
This can and will be done anyway, but not to put at the beck and call of
a self-policing system, but in the service of classical peer review,
upgraded and streamlined for online implementation. (This was discussed
in the Nature online article.)
> Referees would have numbers and each time an author would
> enquire for referees, his name and the title of the article would be
> registered and 3 (or more) referee names would be returned to the author
> after automatic referee number randomisation and combination choice.
Some parametric version of this would be fine, but in the heads of a
respected and capable Editor and Editorial Board, not in the hands of
the authorship at large.
> Referees could establish in this software their availability (for
> example, no more than 4 articles per month) and this would be
> considered by the software during referee choice. Each of these "pool
> of referees" would somehow act as a feedback control center for how many
> times an article was "submitted" and the information would be freely
> accessed by any interested person.
All reasonable suggestions, already being implemented, but in the
service of peer review, not self-policing.
--------------------------------------------------------------------
Stevan Harnad harnad_at_cogsci.soton.ac.uk
Professor of Cognitive Science harnad_at_princeton.edu
Department of Electronics and phone: +44 1703 592-582
Computer Science fax: +44 1703 592-865
University of Southampton
http://www.ecs.soton.ac.uk/~harnad/
Highfield, Southampton
http://www.princeton.edu/~harnad/
SO17 1BJ UNITED KINGDOM
ftp://ftp.princeton.edu/pub/harnad/
Received on Wed Feb 10 1999 - 19:17:43 GMT