Pertinent Prior
Amsci Topic Thread:
"Self-Selected Vetting vs. Peer Review: Supplement or Substitute?" (began November 2002)
Although clinical medical research is not a representative model for research in general (and has led to the wrong-headed idea that the only research that needs to be made Open Access (OA) is what concerns tax-payers' health), it is an instructive model for giving us a gut sense of the importance, even the urgency, of OA.
It is in this light that it is a good idea to ask ourselves, when weighing the adequacy or even the sense of yet another "reform" proposal, to try it out first on articles that concern our health:
We have many times heard the hypothesis that post-hoc vetting by self-selected commentators on the web can serve as a substitute for that pre-evaluation and certification of specialized work by qualified specialists that we call "peer-review." The question to ask yourself about this is whether, if you need to have a loved one treated today, you would like the treatment to be on the basis of (1) unrefereed preprints posted on the web, and possibly/eventually evaluated by possibly-qualified experts -- or you would rather have them treated on the basis of (2) refereed articles that have already been evaluated by qualified experts, and certified by the established quality-standards and track-record of the journal that is answerable for having published them?
I take it that when it comes to loved ones who need treatment today, there is no contest in the mind of anyone who reflects seriously on this question but that (2) is the right answer, with (1) at most only a welcome
supplement to, but certainly no
substitute for (2).
That was the rather shrill family-health version of the question, but it does not require much imagination to see that the answer is the same if we ask it from the viewpoint of a researcher: If you need to decide what finding to invest your limited earthly research time and resources into trying to build upon, is it (1) unrefereed findings posted on the web, possibly/eventually evaluated by possibly-qualified experts, or is it (2) refereed findings already evaluated by qualified experts and certified by the known quality standards of an established journal?
Once again, (1) seems welcome as a supplement to (2), but certainly not as a substitute for it.
I leave it as a lemma for the reader to repeat this exercise, but this time with respect to what papers the overloaded scholar or scientist can afford to spend his finite available reading-time reading: (1) or (2).
Well if it is transparent that anarchic post-hoc self-selected online commentary (1) is not and never will be a substitute for systematic and answerable peer review (2), done and certified in advance, but only a welcome supplement to it, it should not take much more reflection to realize that the most minimal and uninformative aspect of self-selected vetting, namely citation, is even
less suited to take on the a-priori quality-assurance role of peer review.
Again, citation-counts (and other measures of research usage and impact) are welcome post-hoc supplements to research evaluation, but they are certainly no substitute for peer-review itself and its all-important filtering function, certifying in advance what is "safe" for reading, using, applying, consuming. Not being infallible, peer review can use all the extra help it can get from pre- and post-refereeing commentary and citations, but there is no way to bootstrap any of those into performing peer-review's essential and indispensable function in its place.
(Please note that the possibility of posting unrefereed preprints has already made "gate-keeping" an obsolete misnomer for journals: The "gates" they guard are those to their established quality-certification tags, not those to accessing the texts themselves.)
Peer review should never even have come up in the OA context -- except tautologically, in that it is the 2.5 million peer-reviewed articles published in the planet's 24,000 peer-reviewed journals to which OA is meant to maximize access. But somehow, ideas about OA have managed to get entangled with (untested) speculations about peer-review reforms and substitutes. The result is that misconceptions about peer review have been among the panoply of misunderstandings that have already delayed OA (and hence research impact and progress) by at least a decade more than necessary since the day the online medium put 100% OA fully within our reach.
To show that these misconceptions are alive and well in 2005, I quote from an article that appeared today (July 25) in
"The Age": To publish - or to e-publish? By Leslie Cannold
The argument seems to start off well, correctly pointing out that:
LC: "The truth is that academics and universities hold most of the cards in the scholarly publishing game. This is not just because they do the research, write the papers and do the unpaid work required to provide quality assurance by reviewing the work of their peers. It is also because their primary objective is not to profit from the distribution of their work, but to have it read and cited by others."
We expect that the article will now go on to point out that, with the advent of the electronic age, reading and citation can now be maximised by self-archiving the text online. But instead we read:
LC: "In the new electronic age... both the organising of and participation in peer reviews may soon become a thing of the past. Instead of relying on the publisher's reputation and peer review as quality indicators, future scholars may depend on electronic citation counts."
So instead of OA to the peer-reviewed articles, we have OA to unrefereed articles and their citation counts. And now comes the coup de grace:
self-publishing, instead of the self-archiving of peer-reviewed publications; and then, somehow, from out of this vanity press plus citations, something like peer-review is somehow meant to emerge anew:
LC: "[E]very university should have its own serial e-press that would be the first place of publication for first-rate work. Perhaps this press would develop into a number of disciplinary-specific, peer-refereed electronic journals."
We have alas already heard all these ungrounded, untested speculations aired before, and they do not improve with repetition. In essence we are asked to assume -- for no earthly reason -- that the path to OA is to renounce journals, self-publish unrefereed papers in our own institutional vanity presses, count the citations, and wait for peer-review to somehow re-evolve out of all this.
That would already be empty, evidence-free (and almost certainly incoherent) speculation even if there were not a concrete, evidence-based and well-tested alternative staring at us as plainly as the noses on our faces: "You wanted OA to peer-reviewed journal articles? Self-archive them! No need to tamper with either peer review or publication. And self-archiving is not self-publishing: it is just access-provision -- to one's own published articles."
Stevan Harnad
Harnad, S. (1998) The invisible hand of peer review.
Nature (on-line) and
Exploit Interactive.
http://eprints.ecs.soton.ac.uk/2622/
Harnad, S. (2005) Fast-Forward on the Green Road to Open Access: The Case Against Mixing Up Green and Gold.
Ariadne 43.
http://eprints.ecs.soton.ac.uk/10675/
Harnad, S., Brody, T., Vallieres, F., Carr, L., Hitchcock, S., Yves, G., Charles, O., Stamerjohanns, H. and Hilf, E. (2004) The Access/Impact Problem and the Green and Gold Roads to Open Access.
Serials Review 30 (4)
http://eprints.ecs.soton.ac.uk/10209/
