Tuesday, October 17. 2006Premature Rejection Slip![]() Richard Poynder, in "Open Access: death knell for peer review?" has written yet another thoughtful, stimulating essay. But I think he (and many of the scholars and scientists he cites) are quite baldly wrong on this one! What is peer review? Nothing more nor less than qualified experts vetting the work of their fellow specialists to make sure it meets certain established standards of reliability, quality and usability -- standards that correspond to the quality level of the journal whose name and track-record certifies the outcome. Peer review is dynamic and answerable: Dynamic, because it is not just an "admit/eject" decision by a gate-keeper or an "A/B/C/D" mark assigned by a schoolmarm, but an interactive process of analysis, criticism and revision that may take several rounds of successive revisions and re-refereeing. And answerable, because the outcome must meet the requirements set out by the referees as determined by the editor, sometimes resulting in an accepted final draft that is very different from the originally submitted preprint -- and sometimes in no accepted draft at all. Oh, and like all exercises in human judgment, even expert judgment, peer review is fallible, and sometimes makes errors of both omission and commission (but neither machinery nor anarchy can do better). It is also approximate rather than exact; and, as noted, quality-standards differ from journal to journal, but are generally known from the journal's public track record. (The only thing that does resemble an A/B/C/D marking system is the journal-quality hierarchy itself: Meeting the quality-standards of the best journals is rather like receiving an A+, and the bottom rung is not much better than a vanity press.) But here are some other home truths about peer review (from an editor of 25 years' standing who alas knows all too well whereof he speaks): Qualified referees are a scarce, over-harvested resource. It is hard to get them to agree to review, and hard to get them to do it within a reasonable amount of time. And it is not easy to find the right referees; ill-chosen referees (inexpert or biassed) can suppress a good paper or admit a bad one; they can miss detectable errors, or introduce gratuitous distortions. Those who think spontaneous, self-appointed vetting can replace the systematic selectivity and answerability of peer review should first take on board the ineluctable fact of referee scarcity, reluctance and tardiness, even when importuned by a reputable editor, with at least the prospect that their efforts, though unpaid, will be heeded. (Now ask yourself the likelihood that that the right umpires will do their duty on their own, and not even sure of being heeded for their pains.) Friends of self-policed vetting should also sample for a while the raw sludge that first makes its way to the editor's desk, and ask themselves whether they would rather everyone had to contend directly with that commodity for themselves, instead of having it filtered for them by peer review, as now. (Think of it more as a food-taster for the emperor at risk of being poisoned -- rather than as an elitist "gate-keeper" keeping the hoi-poloi out of the club -- for that is closer to what a busy researcher faces in trying to decide what work to risk some of his scarce reading time on, or (worse) his even scarcer and more precious research time in trying to build upon.) And peer-review reformers or replacers should also reflect on whether they think that those who have nothing better to do with their time than to wade through this raw, unfiltered sludge on their own recognizance -- posting their take-it-or-leave-it "reviews" publicly, for authors and users to heed as they may or may not see fit -- are the ones they would like to trust to filter their daily sludge for them, instead of answerable editors' selected, answerable experts. Or whether they would like to see the scholarly milestones, consisting of the official, certified, published, answerable versions, vanish in a sea of moving targets, consisting of successive versions of unknown quality, crisscrossed by a tangle of reviews, commentaries and opinions of equally unknown quality. Not that all the extras cannot be had too, alongside the peer-reviewed milestones: In our online age, no gate-keeper is blocking the public posting of unrefereed preprints, self-appointed commentaries, revised drafts, and even updates and upgrades of the published milestones -- alongside the milestones themselves. What is at issue here is whether we can do without the filtered, certified milestones themselves (until we once again reinvent peer review). The question has to be asked seriously; and if one hasn't the imagination to pose it from the standpoint of a researcher trying to make tractable use of the literature, let us pose it more luridly, from the standpoint of how to treat a family member who is seriously ill: navigate the sludge directly, to see for oneself what's on the market? ask one's intrepid physician to try to sort out the reliable cure from the surrounding ocean of quackery? And if you think this is not a fair question, do you really think science and scholarship are that much less important than curing sick kin? Eppur, eppur... what tugs at me on odd days of the week is the undeniable fact that most research is not cited, nor worth citing, anyway, so why bother with peer review [or, horribile dictu, OA!] for all of that? And on the other end, the few authors of the very, very best work are virtually peerless, and can exchange their masterworks amongst themselves, as in Newton's day. So is all this peer review just to keep the drones busy? I can't say. (I used to mumble things in reply to the effect that "we need to countenance the milk in order to be ensured of the cream rising to the top" or "we need to admit the chaff in order to sift out its quota of wheat" or "we need to screen the gaussian noise if we want to ensure our ration of signal") [1], [2], [3]But I can say that none of this has anything to do with Open Access, now (except that it can be obtruded, along with so many other irrelevant things, to slow OA's progress). If self-archiving mandates were adopted universally, all of this would be mooted. The current peer-reviewed literature, such as it is, would at long-last be OA -- which is the sole goal of the OA movement, and the end of the road for OA advocacy, the rest being about scholars and scientists making use of this newfound bonanza, whilst other processes (digital preservation, journal reform, copyright reform, peer review reform) proceed apace.Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A difficult balance: Peer review in biomedical publication.)Nature 322: 24 - 5. As it is, however, second-guessing the future course of peer review is still one of the at-least 34 pieces of irrelevance distracting us from getting round to doing the optimal and inevitable at long, long last... Harnad, S. (1990) Scholarly Skywriting and the Prepublication ContinuumStevan Harnad American Scientist Open Access Forum Monday, October 16. 2006The Bangalore Commitment: “Self-Archive Unto Others as You Would Have Others Self-Archive Unto You”Appeared inMost of the 2.5 million articles published yearly in our planet's 24,000 research journals are inaccessible to a large portion of their potential users worldwide, but especially in the developing world. One might think that the reason for this is that no research institution can afford to subscribe to all 24,000 journals and that most can only afford a fraction of them -- and this is true, but it is not the whole story, nor the main part of it: For even if all those journals were sold at cost -- not a penny of profit -- they would still remain unaffordable for many of the research institutions worldwide. The only way to make all those articles accessible to all their potential users is to provide “Open Access” to them on the Web, so anyone can access and use them, anywhere in the world, at any time, for free. One could have said the same of food, medicine, and all other human essentials, of course, but one cannot eat digital food or cure diseases with strings of 0's and 1's. Nor, alas, are all the producers of digital products -- let alone of physical food or medicine -- interested in giving away their products for free. So what makes research different (if it is different) and why is it urgent for all of its potential users to have access to it? Research is the source from which future improvements in the quality, quantity and availability of food, medicine, technology, and all other potential benefits to mankind will come, if it is to come at all. And researchers -- unlike the producers of commercial products -- give their findings away: Unlike writers or journalists, researchers do not seek or get fees or royalties for their articles. They give them to their journals for free, and they even mail (and these days email) free copies to any potential user who writes to ask for one. Why do researchers give their articles away for free? Partly for the same reason they are researchers rather than businessmen: They want to make a contribution to knowledge, to research progress. Partly also because that is the nature of the reward structure of science and scholarship: Research is funded, and researchers are employed and paid, on the strength of their "research impact." This used to mean how much they publish, but these days it also means how much their publications are read, used, and built upon, to generate further research and applications, to the benefit of the tax-paying society that funds their research and their institutions. And now we can see why researchers give away their articles and why it is so important that all their potential users should be able to access and use them: Because all access-barriers are barriers to research progress and its benefits (as well as to the advancement of researchers' careers and productivity): If you cannot access a research finding, you cannot use, apply or build upon it. Researchers are not businessmen, but they are not always very practical either. The reason publications need to be counted and rewarded by their employers and funders -- "publish or perish" -- is that otherwise many researchers would just put their findings in a desk drawer and move on to do the next piece of research. (That is part of the price that humanity must pay for nurturing a sector that is curiosity-driven rather than profit-driven.) So, since researchers do need to fund their research and to feed themselves and family, their publications are counted and then rewarded proportionately. But counting publications is not enough: It has to be determined whether the research was important enough to have been worth doing and publishing in the first place; its "research impact" has to be measured: What was its uptake, usage, influence? How many pieces of further research and applications did it generate? Although the measure is crude, and far richer measures are under development, citation counts -- the number of times an article is cited by other articles -- are an indicator of research impact. So, along with publications, citations are counted, in paying researchers and funding their research. And recent studies have shown that the citation counts of articles that are freely available on the web (Open Access) are 25%-250% higher than the citation counts of articles that are only available to those researchers whose institutions can afford a subscription to the journal in which it was published. One would think, in view of these findings, and of the fact that researchers give away their articles anyway, that researchers would all be making their published articles Open Access by now -- by "self-archiving" them in their own institution's online repositories, free for all. Ninety-four percent of journals already endorse self-archiving by their authors. Yet in fact only about 15% of researchers are self-archiving their publications spontaneously today. Perhaps that is about the same percentage of researchers that would be publishing at all, if it were not for the "publish or perish" mandate. So it is obvious what the natural solution is, for research and researchers worldwide, in the online era: the existing publish-or-perish mandate has to be extended to make it into a "publish and self-archive” mandate. International surveys have shown that 95% of researchers would comply with a self-archiving mandate. This has since been confirmed by seven research institutions worldwide (two in Australia, two in Switzerland [one of them CERN], one in Portugal, one in the UK and one in India [National Institution of Technology, Rourkela]) that have already mandated self-archiving: their self-archiving rates are indeed rapidly climbing from the 15% baseline towards 100%. But those are spontaneous institutional mandates, and there are only seven of them so far. There are also a few systematic national mandates: four of the eight UK research funding councils and the Wellcome Trust have now mandated self-archiving. And there are several other national proposals to mandate self-archiving, by the European Commission, a Canadian research council (CIHR) and all of the major US funding agencies (FRPAA). There is no need, however, for developing countries to wait for the developed countries to mandate self-archiving. Developing countries have even more to gain -- for the impact of their own research on the research of others and for their own access to the research of others – because currently both their access and their impact is disproportionately low, relative to their actual and potential research productivity and influence. In the past few years there have been many abstract avowals of support for the Principle of Open Access (e.g., the Bethesda and Berlin and Valparaiso and Goettingen and Scottish and Buenos Aires and Messina and Vienna and Salvador and WSIS and Riyadh Declarations), but these have all merely declared that providing Open Access is a "good thing" and "should be done" -- without saying exactly what should be done, and without committing themselves to doing it! This is rather as if there were a global warming problem, and region after region kept making pious pronouncements to the effect that "something should be done about the global warming problem" instead of affirming that they have actually implemented a concrete emission policy locally, and are now inviting others to do likewise. What the whole world needs now is concrete commitments to the Practice of Open Access. Under the guidance of India’s tireless Open Access advocate, Subbiah Arunachalam, there will be a two day workshop on research publication and Open Access at the Indian Institute of Science in Bangalore on November 2-3, at which representatives from the three most research-active developing countries – India, China and Brazil – will confer in order to frame the “Bangalore Commitment”: a commitment to mandate Open Access self-archiving in their own respective countries and thereby set an example for emulation by the rest of the world: “Self-archive unto others as you would have others self-archive unto you” Stevan Harnad American Scientist Open Access Forum Saturday, October 14. 2006Canada's SSHRC Just Keeps Spinning Its Wheels![]() ![]() Comments on Richard Akerman's blogged notes on David Moorman's SSHRC talk at Institutional Repositories: The Next Generation (Ottawa, October 10, 2006) "SSHRC has embraced OA in principle, but [it's] a big challenge going from principle to action"Five out of the eight UK Research Councils (BBSRC, CCLRC, EPSRC, MRC, NERC) nevertheless seem to have managed to go from principle to action... "Does SSHRC have a policy? No. There is more to this than just mandating OA."Four out of the eight UK Research Councils (BBSRC, EPSRC, MRC, NERC) nevertheless seem to have managed to mandate it, and Canada's CIHR seems to have managed to propose to mandate it... "Figure out how to support OA journals. Conducting experiments to figure out best approach."Why is SSHRC fussing about supporting OA journals instead of mandating the self-archiving of SSHRC research output? Is SSHRC a research funder or a journal funder? The OA mandate pertains to the former, not the latter: to maximizing the access and impact of SSHRC research output, not to the budgeting of SSHRC's journal subsidies. Journals SSHRC may happen to be subsidising have nothing to do with the mandate in question. "look at University of California system"Why not look instead at a system with a mandate, hence one that works -- as voluntarism, demonstrably, does not? "[Create an] SSHRC IR?"An SSHRC Central Repositry (CR)? What on earth for? Mandate that SSHRC fundees self-archive their SSHRC-funded research output in their own institutional IRs. SSHRC does not need a CR of its own in the distributed, interoperable OAI age. "Official Languages Act - websites must be in both languages: how to handle research? 15,000 objects would have to be translated every year"This is such nonsense as to take one's breath away: Can SSHRC not have a library? Are only bilingual or translated journals admissible? Does SSHRC stipulate which language(s) its fundees must publish in? (Besides, SSHRC does not need a CR of its own: Just mandate self-archiving in the fundees' own institutions' IRs.) "grant[s] can't have post-award conditions, e.g. can't require article deposit"Nonsense. If publishing the research in a peer-reviewed journal can be a grant fulfillment condition, so can self-archiving the article. SSHRC is spinning its wheels; it just keeps citing problems in principle while others are busily putting OA into practice. Stevan Harnad"SSHRC Open Access Consultation (Canada)" (Aug 2005) The Special Case of Astronomy
Michael Kurtz (Harvard-Smithsonian Center for Astrophysics) has provided some (as always) very interesting and informative data on the special case of research access and self-archiving practices in Astronomy. His data show that: (1) In astronomy, where all active, publishing researchers already have online access to all relevant journal articles (a very special case!), researchers all use the versions "eprinted" (self-archived) in Arxiv first, because those are available first; but they all switch to using the journal version, instead of the self-archived one, as soon as the journal version is available. That is interesting, but hardly surprising, in view of the very special conditions of astronomy: If I only had access to a self-archived preprint or postprint first, I'd used that, faute de mieux. And as soon as the official journal version was accessible -- assuming that it's equally accessible -- I'd use that. But these conditions -- (i) open accessibility of the eprint before publication, (ii) in one longstanding central repository (Arxiv), for many and in some cases most papers, and (iii) open accessibility of the journal version of all papers upon publication -- is simply not representative of most other fields! In most other fields, (i') only about 15% of papers are available early as preprints or postprints, (ii') they are self-archived in distributed IRs and websites, not one central one (Arxiv), and (iii') the journal versions of many papers are not accessible at all to many of the researchers after publication. That's a very different state of affairs (outside astronomy and some areas of physics). (2) Kurtz's data showing that astronomy journals are not cancelled despite 100% OA are very interesting, but they too follow almost tautologically from (1): If virtually all researchers have access to the journal version, and virtually all of them prefer to use that rather than the eprint, it stands to reason that it is not being cancelled! (What is cause and what is effect there is another question -- i.e., whether preference is driving subscriptions or subscriptions are driving preference.) (3) In astronomy, as indicated by Kurtz, there is a small, closed circle of core journals, and all active researchers worldwide already have access to all of them. But in many other fields there is not a closed circle of core journals, and/or not all researchers have access. Hence access to a small set of core journals is not a precondition for being an active researcher in many fields -- which does not mean that lacking that access does not weaken the research (and that is the point!). (4) I agree completely that there is a component of self-selection Quality Bias (QB) in the correlation between self-archiving and citations. The question is (4a) how much of the higher citation count for self-archived articles is due to QB (as opposed to Early Advantage, Competitive Advantage, Quality Advantage, Usage Advantage, and Arxiv (Central) Bias)? And (4b) does self-selection QB itself have any causal consequences (or are authors doing it purely superstitiously, since it is has no causal effects at all)? The effects of course need not be felt in citations; they could be felt in downloads (usage) or in other measures of impact (co-citations, influence on research direction, funding, fame, etc.). The most important thing to bear in mind is that it would be absurd to imagine that somehow OA guarantees a quality-blind linear increment to the usage of any article, regardless of its quality. It is virtually certain that OA will benefit the better articles more, because they are more worth using and trying to build upon, hence more handicapped by access-barriers (which do exist in fields other than astro). That's QA, not QB. No amount of accessibility will help unciteable papers get used and cited. And most papers are uncited, hence probably unciteable, no matter how visible and accessible you may try to make them! (5) I think we agree that the basic challenge in assessing causality here is that we have a positive correlation (between proportion of papers self-archived and citation-counts) but we need to analyze the direction of the causation. The fact that more-cited papers tend to be self-archived more, and less-cited papers less is merely a restatement of the correlation, not a causal analysis of it: The citations, after all, come after the self-archiving, not before! The only methodologically irreproachable way to test causality would be to randomly choose a (sufficiently large, diverse, and representative) sample of N papers at the time of acceptance for publication (postprints -- no previous preprint self-archiving) and randomly impose self-archiving on N/2 of them, and not on the other N/2. That way we have random selection and not self-selection. Then we count citations for about 2-3 years, for all the papers, and compare them. No one will do that study, but an approximation to it can be done (and we are doing it) by comparing (a) citation counts for papers that are self-archived in IRs that have a self-archiving mandate with (b) citation counts for papers in IRs without mandates and with (c) papers (in the same journal and year) that are not self-archived. Not a perfect method, problems with small Ns, short available time-windows, and admixtures of self-selection and imposed self-archiving even with mandates -- but an approximation nonetheless. And other metrics -- downloads, co-citations, hub/authority scores, endogamy scores, growth-rates, funding, etc. -- can be used to triangulate and disambiguate. Stay tuned. Now some comments: On Tue, 10 Oct 2006, Michael Kurtz wrote: "Recently Stevan has copied me on two sets of correspondance concerning the OA citation advantage; I thought I would just briefly respond to both.And it also shows how anomalous Astronomy is, compared to other fields, where it is certainly not true that every researcher has subscriptions to the main journals... "Figure 5 of the J Electronic Publishing paper also shows that there is no effect of cost on the OA reads (and thus by extension citation) differential. Note in the plot that there is no change in slope for the obsolescence function of the reads (either of preprinted or non-preprinted) at 36 months. At 36 months the 3 year moving wall allows the papers to be accessed by everyone, this shows clearly that there is no cost effect portion of the OA differential in astronomy. This confirms the conclusion of my IPM article."And it underscores again, how unrepresentative astronomy is of research as a whole. "Citations are probably the least sensitive measure to see the effects of OA. This is because one must be able to read the core journals in order to write a paper which will be published by them. It is really not possible for a person who has not been regularly reading journal articles on, say, nuclear physics, to suddenly be able to write one, and cite the OA articles which enabled that writing. It takes some time for a body of authors who did not previously have access to form and write acceptable papers."In astronomy -- where the core journals are few and a closed circle, and all active researchers have access to them. But this is not true of research as a whole, across disciplines (or around the world). Researchers in most fields are no doubt handicapped for having less than full access, but that does not prevent them from doing and publishing research altogether. "Any statistical analysis of the causal/bias distinction must take into account the actual distribution of citations among articles. This is why I made the monte carlo analysis in the IPM paper. As a quick example for papers published in the Astrophysical Journal in 2003: The most cited 10% have 39% of all citations, and are 96% in the arXiv; the lowest cited 10% have 0.7% of all citations and are 29% in the arXiv. Showing the causal hypothesis is true will be very difficult under these conditions."(i) Since all of the published postprints in all these journals are accessible to all research-active astronomers as of their date of publication, we are of necessity speaking here mostly about an Early Access effect (preprints). Most of the other components of the Open Access Advantage (Competitive Advantage, Usage Advantage, Quality Advantage) are minimized here by the fact that everything in astronomy is OA from the date of publication onward. The remaining components are either Arxiv-specific (the Arxiv Bias -- the tradition of archiving and hence searching in one central repository) or self-selection [Quality Bias] influencing who does and does not self-archive early, with their prepublication preprint. Since most fields don't post pre-refereeing preprints at all, this comparison is mostly moot. For most fields, the question about citation advantage concerns the postprint only, and as of the date of acceptance for publication, not before. (ii) In other fields too, there is the very same correlation between citation counts and percentage self-archived, but it is based on postprints, self-archived at publication, not pre-refereeing preprints self-archived much earlier. And, most important, it is not true in these fields that the postprint is accessible to all researchers via subscription: Many potential users cannot access the article at all if it is not self-archived -- and that is the main basis for the OA impact advantage. "Perhaps the journal which is most sensitive to cancellations due to OA archiving is Nuclear Physics B; it is 100% in arXiv, and is very expensive. I have several times seen librarians say that they would like to cancel it. One effect of OA on Nuclear Physics B is that its impact factor (as we measure it, I assume ISI gets the same thing) has gone up, just as we show in the J E Pub paper for Physical Review D. Whether Nuclear Physics B has been cancelled more than Nuclear Physics A or Physics Letters B must be well known at Elsevier."It is an interesting question whether NPB is being cancelled, but if it is, it clearly is not because of self-archiving, nor because of astronomy's special "universal paid OA" OA to the published version: if NPB is being cancelled, it is for the usual reason, which is that it is not good enough to justify its share of the institution's journal budget. Harnad, S. (2005) OA Impact Advantage = EA + (AA) + (QB) + QA + (CA) + UAStevan Harnad American Scientist Open Access Forum Thursday, October 12. 2006CIHR Proposes 99.99% Optimal OA Self-Archiving Mandate![]() Canadian Institutes of Health Research (CIHR) has proposed a 99.99% optimal Open Access Self-Archiving Mandate: There is only one unnecessary and confusing clause in CIHR's policy: (2b). (2b) is redundant with [1]! (2b) says the author must publish in a journal that allows [1]. But that is already implicit in [1] -- it is not a sub-option of [2]. [1] is the requirement to self-archive immediately (and to set access as Open Access within 6 months). Alternative [2] is to publish in an Open Access journal. That covers all the alternatives! (2b) is completely redundant. So (2b) should simply be dropped. That's all, really. There are still a few minor changes that would make the policy simpler, clearer, and more systematic and coherent. In order to encourage a uniform practice that will generalize and apply to all fields, whether or not funded by CIHR, it would be best if CIHR's uniform rule consisted of just these 5 components: I. must deposit final peer-reviewed manuscript (or published version)This way, everything gets deposited immediately, and access is OA within 6 months. The IR should be the preferred default locus, from which PubMed Central or other archives can harvest, but direct deposit elsewhere can be allowed as an option if the researcher has no institutional IR yet. ![]() (CIHR also requires making research data and materials available for reasonable requests: Might as well recommend -- but not require -- that they are self-archived too, wherever possible!) Bravo CIHR! Stevan Harnad American Scientist Open Access Forum PS: Note that, unlike the Wellcome Trust's Self-Archiving Mandate, CIHR's proposed mandate does not offer to fund option (2a) (publishing in an Open Access or hybrid "Open Choice" journal). Apparently CIHR did not feel it had the spare cash for this. This is quite understandable (although no doubt some publishers will complain vociferously about it): The fact is that all potential publication funds are currently tied up in covering the costs of institutional subscriptions, worldwide. If and when self-archiving should ever lead to institutional subscription cancellations that make the subscription model unsustainable, then those very institutional windfall savings themselves will be the natural source for the cash to cover OA publishing costs. No need to take it from research funds at this time, when it is unaffordable. OA is the immediate and urgent (and long-overdue) priority today, not pre-emptively cushioning a hypothetical transition to another publishing cost-recovery model (except where the spare cash is available). Please note that a public consultation has been launched to seek comments on CIHR's proposed Policy on Access to Research Outputs. Tuesday, October 10. 2006Hypotheses Non FingoJ.W.T.Smith (Templeton Library, University of Kent)Hypotheses non fingo. There is no "Harnad model": Research is published in c. 24K peer-reviewed journals (c. 2.5M articles annually). (Datum, not hypothesis.) Not all would-be users can access all those articles online. (Datum, not hypothesis.) Self-archiving supplements access, for those would-be users. (Datum, not hypothesis.) Self-archiving is correlated with higher and earlier download and citation impact. (Datum, not hypothesis.) Self-archiving is explicitly endorsed by 93% of journals. (Datum, not hypothesis.) Only c. 15% of annual articles are being spontaneously self-archived today. (Datum, not hypothesis) 95% of researchers surveyed report they will self-archive if it is mandated. (Datum, not hypothesis.) When self-archiving is mandated, it rapidly rises toward 100%. (Datum, not hypothesis.) No evidence has been reported to date that self-archiving causes cancellations. (Datum, not hypothesis.) Hypotheses non fingo. There is no "Harnad model."[*Self-archiving might (or might not) eventually cause cancellations and a change in journal publishing model. (Hypothesis) Mea maxima culpa!] Stevan HarnadBerners-Lee, T., De Roure, D., Harnad, S. and Shadbolt, N. (2005) American Scientist Open Access Forum Monday, October 9. 2006Critique of EPS/RIN/RCUK/DTI "Evidence-Based Analysis of Data Concerning Scholarly Journal Publishing"
UK scholarly journals: 2006 baseline report An evidence-based analysis of data concerning scholarly journal publishing. Prepared on behalf of the Research Information Network, Research Councils UK and the UK Department of Trade and Industry. By Electronic Publishing Services Ltd In association with Professor Charles Oppenheim and LISU at Loughborough University Department of Information Science This is a rather long and repetitious report, but it does contain a few nuggets. It is obviously biassed, but biassed in a restrained way, meaning it does not really try to conceal its biases, nor does it overstate biassed conclusions. It also (reluctantly, but in most cases candidly) acknowledges its own weaknesses. (The Report was commissioned by RIN, RCUK and DTI, but it is glaringly obvious that the questions, answers and interpretations have been slanted toward the interests of the publishing lobby rather than those of the research community -- possibly because the research community has no lobby in this matter, apart from the OA movement itself! Nevertheless, there has been considerable circumspectness, at least in the summary and conclusion passages, with weak points and gaps usually pointed out explicitly rather than denied or concealed, and with the overall preoccupation with publishing interests rather than research interests very open too.) Some quotes and comments: Whilst some evidence does suggest that [self-archiving in] repositories [is] an important new factor in the journal cancellation decision process, and one which is growing in significance, there is no research reporting actual or even intended journal subscription cancellation as a consequence of the growth of OA self-archived repositories.So far, this sounds fair and reasonable. (In fact, this is the gist of the Report! The rest is mostly special pleading.) Subscriptions are reported to have been declining over a period of 10+ years, but for a number of reasons. Proving or disproving a link between availability in self-archived repositories and cancellations will be difficult without long and rigorous research. In this connection, the outcome of research recently announced by the Research Councils UK (RCUK) with the co-operation of Macmillan, Blackwell and Elsevier, will be eagerly awaited, even though a report is not due until late 2008.With evidence of self-archiving's benefits to research mounting, and zero evidence yet of any negative effect at all on publisher revenue, publishers nevertheless seem quite willing to wait (and keep research waiting too), trying to fend off self-archiving and its potential benefits to research for a long time to come yet, in order to keep trying to find some evidence of negative causal effects on publisher revenue (or, failing that, to deny positive causal effects on research impact). Note that whereas a link between OA self-archiving and subscription decline has not yet been "proved or disproved" (not for want of looking!) -- and it is for that reason that we are hearing these calls for "long and rigorous research" -- the vast preponderance of the evidence we do have has already "proved" a "link" between OA self-archiving and citation counts (a link that is almost certainly causal, despite the wishful thinking of some who have a vested interest in its all turning out to be merely a-causal self-selection and superstition on the part of authors). The question that the research community accordingly needs to ask itself is whether self-archiving's evidence-based benefits to research should be held in abeyance still longer, and meanwhile interpreted by default as a-causal, in order to buy still more time to try to "prove/disprove" hypothetical subscription declines for which there is no evidence whatsoever to date, even in fields where self-archiving has been near 100% for years. (Researchers should also go on to ask themselves whether the research benefits should be held in abeyance even if they are causally linked to a subscription decline: Is research impact to be sacrificed in the service of publisher revenue? Are we conducting and funding research in order to generate -- or to safeguard -- publisher revenue?) There is no evidence as yet to demonstrate any relationship (or lack of relationship) between subscription cancellations and repositories. Work in this field would need sufficient, representative and balanced samples, and the collaboration of all stakeholders, including especially research institutions and publishers. Any such study will need to be maintained over a fairly extended period, with regular reports, since it seems likely that the position could change with time if the contents of self-archiving repositories become progressively more comprehensive.This would be fine, if proposed as an extended research project to be conducted after self-archiving mandates are in place, to analyze their long-term effects on subscriptions. But this would be an exceedingly self-serving suggestion on the part of the publishing community (and a methodologically empty one) if meant as a "pilot" study that must somehow be conducted before adopting self-archiving mandates. (And it would be exceedingly self-defeating of the research community to even consider accepting such a pre-emptive suggestion as a precondition, before adopting self-archiving mandates.) There is some consistency in results that show more citations for articles self-archived in repositories as distinct from the same or similar articles available [only via journal] subscription (although there have also been a few contradictory results). Overall, deposit of articles in open access repositories seems to be associated with both a larger number of citations, and earlier citations for the items deposited.This a fair summary -- except that immediately after stating it, this "association" is about to be deconstructed (much as the "association" between cigarette-smoking and lung cancer was deconstructed for years and years by the tobacco industry, claiming that only correlation had been demonstrated, and not causation). Read on: The reasons for this [association] have not been clearly established - there are many factors that influence citation rates, including the reputation of the author, the subject-matter of the article, the self-citation rate, and, of course, how important or influential the repository is in its own right. The little existing evidence suggests that a possible [sic] reason for increased citation counts is not that the materials were free, or that they appeared more rapidly, but that authors put their best work into OA format. This research was limited to one discipline, however [astronomy], and more extensive evidence is required to validate this finding.This (important) study by Kurtz et al in astronomy, however, is not what the vast majority of the evidence (no longer little!) shows: Moreover, as noted, this a-causal interpretation -- only one of the possible interpretations of the astronomy evidence -- also happens to be the interpretation that the publishing community prefers for all the self-archiving evidence, in all fields. The alternative interpretation is that the relationship is causal: that the OA advantage is not merely an arbitrary whim on the part of the better authors to make their work OA, to no causal effect at all (why on earth would they be doing it at all then?): They do it because making their work more accessible increases its accessibility, uptake, downloads, usage, applications, citations, impact -- exactly as the correlational evidence shows, without exception, in field after field. (NB: The only methodologically unexceptionable way to demonstrate causation here, by the way, is to select a large enough random sample of articles, divide them in half randomly, mandate half of them to be self-archived and half not, and then compare their respective citation counts after a few years. No one is likely to do quite that study -- any more than it was likely that a large random sample of people would be divided in half randomly, with half mandated to smoke and half not! But we are in the process of doing an approximation to that causal study, by comparing the citation counts of articles in the IRs of the (few) institutions that have already mandated self-archiving with the average for other articles in the same journals/years in which those articles appeared, but that have not been self-archived; we will also compare the size of the OA advantage for mandated and comparable non-mandated self-archiving. [We do not believe for a moment that these data are necessary to demonstrate causation, as causation is a virtual certainty anyway, but we are ready to play the game, in order to try to cut short the absurd delay in doing the obvious: mandating self-archiving universally.]) Although quite a lot of evidence has been collected regarding the quantitative effect of OA on citation counts (whether in the form of OA journals or as self-archived articles), much of it is scattered, uses inconsistent methods and covers different subject areas.Yet, despite this scatter, methodological inconsistency and diversity, virtually all of it keeps showing exactly the same consistent pattern: A citation (and download) advantage for the OA articles. (No amount of special pleading can make that stubborn pattern go away!) Consistent longitudinal data over a period of years to measure IF trends in a representative range of journals would fill this gapThere is no gap! There is a growing body of studies, across all fields and all journals, that keeps showing exactly the same thing: the OA advantage (in article citations and article downloads: this is not about journal impact factors, especially because comparing different journals is comparing apples and oranges). (There seems to be a confusion here between the existence of the correlation itself, between self-archiving and citation count counts -- this is found consistently, over and over -- and the question of the causal relation, which will not be answered by longtitudinal data (we have longtitudinal data already!) but by comparing mandated and unmandated self-archiving: if they both show the OA advantage, then the effect is causal and self-selection bias is a minor component.) e.g., studying a range of journals that were toll-access and went OA (or vice versa). In the short-term, more data in different disciplines measuring the impact on citation counts of articles in hybrid journals or articles that are available in both forms versus articles that are only available in one of the forms will improve the evidence base.No, the question about the reality and causality of the OA advantage will not be settled by OA journal vs. non-OA journal comparisons; that can always be dismissed as comparing apples with oranges, and, failing that, can always be attributed to self-selection bias (i.e., choosing to publish one's better work in an OA journal)! And if we wait for the uptake of hybrid Open Choice -- i.e., paying the journal to self-archive the published PDF for you -- these "longtitudinal" studies are likely to take till doomsday (and any positive outcome can still be dismissed as self-selection bias in any case!). What is needed is precisely the data already being gathered, on huge samples, across all disciplines, comparing citation counts for self-archived versus non-self-archived articles within the same journal and year. The result has been a consistent, high OA Advantage (which has elicited a lot of special pleading about causality). So we will look at the mandated subset of the self-archived papers, to try to show that the OA advantage is not (only, or mostly) a self-selection effect (Quality Bias [QB]). (There is undoubtedly a non-zero self-selection [QB] component in the OA advantage, but there are many other components as well, including a Quality Advantage [QA], an Early Access Advantage [EA], a Competitive Advantage [CA, which will, like QB, vanish once all articles are OA], and a Usage (Download) Advantage [UA]. At 100% OA, there will no longer be any QB or CA (or Arxiv Advantage [AA]), but EA, QA and UA will still be going strong. EA and UA components have already been confirmed by the Kurtz study in astronomy. QA is implied by the repeated finding of a positive correlation between citation count and the proportion of those articles with that citation count that are OA. The mandate study will try to show that this correlation is causal, i.e., QA, not QB.) First, the issue is article citation counts, not journal Impact Factors (IFs).Harnad, S. (2005) OA Impact Advantage = EA + (AA) + (QB) + QA + (CA) + UA.The whole area of the relationship between citation counts and scholarly communication channels is confused because of problems associated with quality bias [QB] (e.g., if scholars tend to self-archive only their best work, as suggested by Kurtz et al. [in astronomy]; alternatively, it may be that only the best journals are OA). In other words, differences in citation counts and IFs may simply reflect the quality of the materials under study rather than having anything to do with the channel by which the material is made available. Second, this is all special pleading. The biggest OA effects are based on comparing articles within the same journal/year. The size of the effect is indeed correlated with the quality of the article, because no amount of accessibility will generate citations for bad articles, whereas good articles benefit the most from a level playing field, with all affordability/accessibility barriers removed: that is the Quality Advantage [QA]. The idea that the Quality Advantage is merely a Quality (Self-Selection) Bias [QB], i.e., that the advantage is merely correlational, not causal, is of course a logical possibility, but it is also highly improbable (and would imply that accessibility/affordability barriers count for nothing in usage and citations, and that the better work is being made OA by its authors for purely superstitious reasons, because doing so has no effect at all!). Overall, we concur with Craig's introduction that "the problems with measuring and quantifying an Open Access advantage are significant. Articles cannot be OA and non-OA at the same time."They need not be. It is sufficient if we take a large enough sample of articles that are OA and non-OA from the same journals and years. Randomly imposing the self-archiving would be the only way to equate them completely (and our ongoing study on mandated self-archiving will approximate this). (The analysis by Craig, commissioned by Blackwell Publishing, has not, so far as I know, been published.) "Further, the variation of citation counts between articles can be extremely high, so making controlled comparisons of OA vs. non-OA articles nigh on impossible" [Craig, Blackwell Publishing](The way Analysis of Variance works is to compare variation between and within putatively different populations, to determine the probability that they are in reality the same population. The published comparisons show that the OA/non-OA differences are highly significant, despite the high variance.) It would of course be absurd to try to compare citation counts for OA and non-OA articles having the same citation counts. But we can compare OA and non-OA article counts among articles having the same citation counts, in the same journals -- and what we find is a strong positive correlation between the citation count and the proportion of articles that are OA (just as Lawrence reported in 2001, but not only in computer science, but across all 12 disciplines studies so far, and with much bigger sample sizes): Note that the appendix to the Report under discussion here, states, in connection with the above study, which it cites:Source 4.8: Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year Cross-Disciplinary Comparison of the Growth of Open Access and How it Increases Research Citation Impact. IEEE Data Engineering Bulletin 28(4) pp. 39-47. "Harnad is THE advocate of OA and, thus, whilst expert in the field, is inevitably biased."There is a bit of irony in the fact that in connection with another of the studies it cites: the appendix of the Report goes on to say:Source 4.9: Harnad, S, Brody, T, Oppenheim, C et al, Comparing the impact of open access versus non open access articles in the same journals, D-Lib Magazine, 10,(6), 2004, "Harnad is THE exponent of OA, but, thus, potentially less objective."Ironic (or, shall we say, conflicted, since this Report aspires to be a neutral one as between the interests of the research community and the publisher community), because the sole named collaborator on the Report is also a co-author of the above-cited study! Let us agree that we all have views on the underlying issues, but that reliable data speak for themselves, qua data, and our data (and those of others) keep showing the same consistent OA Advantage. The disagreement is only on the interpretation: whether or not the consistent correlations are causal. And here, allegiances are tugging on both sides: Those favouring causality tend to come from the research community, those favouring a-causality tend to come from the publishing community. (Let us hope that the data from mandated self-archiving will soon settle the matter objectively.) "[since] any Open Access advantage appears to be partly [sic] dependent on self-selection, the more articles that are {self-}archived... you'd expect to see any Open Access advantage reduce." [Craig, Blackwell Publishing]Note that Craig carefully says "partly" -- and that we agree that self-selection is one of the many potential contributors to the OA advantage. We also agree, of course, that once 100% OA is reached, the OA citation advantage -- in the form of an advantage of OA over concurrent non-OA articles -- will be reduced: indeed it will vanish! With all articles OA, there can no longer be either a Competitive Advantage [CA] or a Self-Selection Advantage (Quality Bias, QB) of OA over (non-existent) non-OA. But the Quality Advantage [QA] will remain. (Higher quality articles will be used and cited more than they would have been if they had not been OA: this is not a competitive advantage but an absolute one.) And the Early Advantage [EA] as well as the Usage (Download) Advantage [UA] will remain too (as already shown by Kurtz's findings in Astronomy). "Authors self-archiving in the expectant belief that each and every paper they archive will receive an Open Access advantage of several hundred percent are going to be sorely disappointed." [Craig, Blackwell Publishing]This too is correct, but who on earth thought that OA would guarantee that all work would be used, whether or not it was any good? OA levels the playing field so merit can rise to the top, unconstrained by accessibility or affordability handicaps. But bad remains bad, and let's hope that researchers will continue to avoid trying to build on weak or invalid findings, whether or not they are OA. The OA advantage is an average effect, not an automatic bonus for each and every OA article; moreover, the OA advantage is highly correlated with quality: The higher the quality, the higher the advantage. It is this effect that is open to the a-causal interpretation that the Quality Advantage [QA] is merely a Quality Bias [QB] (Self-Selection). But, equally (and, in my view, far more plausibly) it is open to the causal interpretation that OA causes wider usage and citation precisely because it removes all accessibility/affordability constraints that are currently limiting uptake and usage. That does not mean everything will be used more, regardless of quality ("usefulness"): But it will allow users (who are quite capable of exercising self-selection too!) to access and use the better work, selectively. In addition, since the distribution of citations is not gaussian -- a small percentage of articles receives most of the citations and more than half of articles receive no citations at all -- it is almost axiomatic that the OA advantage will be strongest in the high-quality range Finally, it is worth noting that all researchers in the field are agreed that if the vast majority of scholarly publications become available in OA form, no citation advantage to OA will be measurable.It is a tautology that with 100% OA, the OA/NOA ratio is undefined! But EA will still be directly measurable, and it will be possible to infer UA and QA indirectly (UA by comparing downloads for articles of the same age, before and after OA for the same articles, and QA by doing the same with citations; the Kurtz study used such methods in Astronomy. But by that time (100% OA), not many people will still have any interest in the a-causal hypothesis. Thus, what OA advantage there is will prove to be temporary if OA does become the standard mode of publication.This, however, is simply incorrect. At 100% OA, the Competitive Advantage (CA) will be gone; the Self-Selection Advantage (Quality Bias, QB) will be gone; the method of comparing citation counts for OA and non-OA articles within the same journal and year will be gone. So much is true by definition. But (as Kurtz has shown in Astronomy), the Early Advantage and the Usage Advantage will still be there. And the Quality Advantage, will still be there too; and that was what this was all about: Not just a horse-race for who can make his articles OA first, so as to reap the competitive advantage before 100% OA is reached (though that's not a bad idea!); not a guarantee that, no matter how bad your work, you can increase your citations by making them OA; but a guarantor that with access-barriers removed, quality will have the best chance to have its full potential impact, to the benefit of research productivity and progress itself, as well as the authors, institutions and funders of the high quality work. (There is a bit of a [lurid] analogy here with saying that if only we can get everyone to smoke, it will be clear that smoking has no differential effects on human health! Perhaps the converse is a better way to look at it: if only we could get everyone to stop smoking, smoking will no longer have a differential effect on human health!) (PS: OA is not a "mode of publication": OA publication is a mode of publication. OA itself is a mode of access-provision, which can be done in two ways, via OA publication or via OA self-archiving of non-OA publications.) Self archived articlesLet us be clear: The many OA vs. non-OA studies, ours and everyone else's, across more than a dozen different disciplines, many of them based on large-scale samples, all show the very same consistent pattern of positive correlation between OA and citation counts. Those are data, and they are not under dispute. The only "claim" under dispute is that that consistent correlation is causal... Antelman (Source 4.1) is arguably the most carefully constructed study of the question. Articles in four disciplines were evaluated, and in each case it was found that open access articles had greater citation counts than non-open access articles.One wonders why this particular small-scale study (of about 2000 articles in 4 fields) was singled out, but in any event, it shows exactly the same pattern as all the other studies (some of them based on hundreds of thousands of articles instead of just a few thousand, in three times as many fields). Eysenbach challenges the notion that OA "green" articles (i.e., those in repositories) are more effective than OA "gold" (i.e., those published in OA journals, such as those produced by Public Library of Science) in obtaining high citation counts. It is this part of his paper that produced a furious response from Harnad, much of it focused on particular details.The issue was not about OA green (self-archived) articles producing higher citation counts than OA gold (OA-journal)! No one had claimed one form of OA was more effective than the other in generating the OA Advantage before the Eysenbach study: It was Eysenbach who claimed to have shown gold was more effective than green -- indeed that green was only marginally effective at all! And I think anyone reading the exchanges will see that all the fury is on the Eysenbach side. All I do is point out (rather patiently) where Eysenbach is overstating or misstating his case: Harnad, S. (2006)PLoS, Pipe-Dreams and Peccadillos PLoS Biology eletters (May 16, 2006) [1] [2] [3] [4]Eysenbach's study does find the OA advantage, as many others before it did. It certainly doesn't show that the gold OA advantage is bigger than the green OA advantage, in general. It simply shows that for the 1500-article sample in the one journal tested, Proceedings of the National Academy of Sciences (PNAS), a very high impact journal, both paid OA (gold) and green OA (free) increased citation counts over non-OA, but gold increased them more than green. That result is undisputed. Its extrapolation to other journals is: The likely explanation of the PNAS result is very simple: PNAS is not a randomly chosen, representative journal: it is a very high-impact, very high visibility, interdisciplinary journal, one of very few like it (along with Nature and Science). Articles that pay for OA are immediately accessible at PNAS's own high-visibility website -- a website that probably has higher visibility than any single institution's IR today. So PNAS articles made freely accessible at PNAS's website get a bigger OA advantage than PNAS articles made made freely accessible by being self-archived in the author's own IR. The reason it definitely does not follow from this that gold OA is bigger than green OA is very simple: Most journals are not PNAS, and do not have the visibility or average impact of PNAS articles! Hence Eysenbach's valid finding for one very high-impact journal simple does not generalize to all, most, or even many journals. Hence it is not a gold/green effect at all, but merely a very high-end special case. Apart from the spurious gold/green advantage, Eysenbach did confirm, yet again, (1) the OA advantage itself, and confirmed it (2) within a very short time range. These are both very welcome results (but not warranting to be touted, as they were, by both the author and by the accompanying PLoS editorial, as either the first "solid evidence" of the OA advantage -- they certainly were not that -- or a demonstration that gold OA generates more citations than green OA: the very same method has to be tried on middle and low-ranking journals too, before drawing that conclusion!). (Nor are the PLoS/PNAS results any more exempt from the methodological possibility of self-selection bias [QB] than any of the many prior demonstrations of the OA advantage, as authors self-choose to pay PNAS for gold OA as surely as they self-choose to self-archive for green OA!) The fury on Eysenbach's part came from my pointing out that his and PLoS's claim to primacy for demonstrating the OA advantage (and their claim of having demonstrated a general gold-over-green advantage) was unfounded (and might have been due to both PLoS's and Eysenbach's zeal to promote publication in gold journals: Eysenbach is the editor of one too, but not a high-end one like PNAS or PLoS): Eysenbach's was just the latest in a long (and welcome) series of confirmations of the OA advantage (beginning with Lawrence 2001), the prior ones having been based on far larger samples of articles, journals and fields (and there was no demonstration at all of a general gold over green advantage: just the one non-representative, hence non-generalisable special case of PNAS). Both authors believe that OA produces a citation advantage, but Eysenbach has presented evidence that casts doubt on Harnad's notion that the "green" route is the preferred route to getting that increased impact.Green may not be the preferred route to OA for editors of gold journals, but it is certainly the preferred route for the vast majority of authors, who either have no suitable gold journal to publish in, or lack the funds (or the desire) to pay the journal to do what they can do for free for themselves. The only case in which paid gold OA may bring even more citations than free green OA (even though both increase citations) is in the very highest quality journals, such as PNAS, today -- but that high-end reasoning certainly does not generalise to most journals, by definition. (And it will vanish completely when OA self-archiving is mandated, and the harvested IR contents become the locus classicus to access the literature for those whose institutions are not subscribed to the journal in which a particular article appeared -- whether or not it is a high-end journal.) (There is also a conflation of the (less interesting) question of (1) whether green or gold generates a greater OA citation advantage [answer, for high-end journals like PNAS, gold does, but in general there is no difference] with the (far more important) question of (2) whether green or gold can generate more OA [answer: green can generate far more OA, far more quickly and easily, not just because it does not cost the author/institution anything, but because it can be mandated without needing either to find the extra funds to pay for it or to constrain the author's choice of which journal to publish in]. However, despite the intuitive attractiveness of the hypothesis that OA will lead to increased citations because of easier availability, the one systematic study of the reasons for the increased citations - by Kurtz (Source 4.12) - showed that in the field of astronomy at least, the primary reason was not that the materials were free, or that they appeared more rapidly, but that authors put their best work into OA format, and this was the reason for increased citation counts.Astronomy is an interesting but anomalous field: It differs from most other fields in that: (1) Astronomy consists of a small, closed circle of journals. (2) Virtually all research-active astronomers (so I am told by the author) have institutional access to all those journals. (3) For a number of years now, that full institutional access has been online access. (4) So astronomy is effectively a 100% OA field. (5) Hence the only room left for a directly measurable OA advantage in astronomy is (5a) to self-archive the paper earlier (at the preprint stage) [EA] or (5b) to self-archive it in Arxiv (which has evolved into a common central port of call, so it generates more downloads and citations -- mostly at the preprint stage, in astronomy). (6) What Kurtz found, was that under these conditions, higher quality (higher citation-count) papers were more likely to be self-archived. (7) This might be a quality self-selection effect (QB) (or it might not), but it is clearly occurring under very special conditions, in a 100% OA field. (8) Kurtz did make another, surprising finding, which has bearing on the question of how much of a citation advantage remains once a field has reached 100% OA. (9) By counting citations for comparable articles before and after the transition to 100% OA, Kurtz found that the citations per article had actually gone down (slightly) rather than up, with 100% OA. (10) But a little reflection suggests a likely explanation: This slight drop is probably a shift in balance with a level playing field: (11) With 100% OA (i.e., equal access to everything), authors don't cite more articles, they cite more selectively, able now to focus on the best, most relevant work, and not just on the work their institutions can afford to access. (12) Higher quality articles get more citations, but lower quality articles of which there are far more (some perhaps previously cited by default, because of accessibility constraints) are cited less. (13) On balance, total citations are slightly down, on this level playing field, in this special, small, closed-circle field (astronomy), once it reaches 100% OA. (14) It remains to be seen whether total and average citations go up or down when other fields reach 100% OA. (15) What Kurtz does report even in astronomy is that although total citations are slightly down, downloads are doubled. (16) Downloads are correlated with later citations, but perhaps at 100% OA this is either no longer true, or true only for higher quality articles. Similarly, more carefully conceived work on the impact of both OA journals and self-archiving on the quality of research communications, especially on the peer review system, will be required.OA journals are peer-reviewed journals: What sort of impact are they feared to have on peer review? And why on earth would the self-archiving of peer-reviewed, published postprints have any impact on the peer review system? The peers review for free. (Could this be just a veiled repetition of the question about the impact of self-archiving on journal revenues, yet again?) Recently, the results of a study undertaken by Ware for ALPSP, which were published in March 2006 (Source 1.16, in Area 1), have provided at least some initial data on the question of the possible linkage between the availability of self-archived articles in an OA repository and journal subscription cancellations by libraries...: availability of articles in repositories was cited as either a "very important" or an "important" possible factor in journal cancellation by 54 per cent of respondents, even though ranking fourth after (i) decline of faculty need, (ii) reduced usage, and (iii) price. When respondents were invited to think forward five years, availability in a repository was still fourth-ranking factor, but the relevant percentage had risen to 81. Whilst this is not evidence of actual or even intended cancellation as a consequence of the growth of OA self-archiving repositories, it strongly suggests that such repositories are an important new factor in the decision process, and growing in significance.Summary: No evidence of cancellations, but speculations by librarians to the effect that their currently fourth-ranking factor in cancellations might possibly become more important in the next five years... Sounds like sound grounds for fighting self-archiving mandates and trying to deny research the benefit of maximized impact for yet another five years -- if one's primary concern is the possible impact of mandated self-archiving on publishers' revenue streams. But if one's primary concern is with the probable impact of mandated self-archiving on research impact, this sort of far-fetched reasoning has surely earned the right to be ignored by the research community as the self-serving interference in research policy that it surely is. Stevan Harnad American Scientist Open Access Forum Friday, October 6. 2006Responses to EC Self-Archiving Mandate Recommendation![]() Stevan Harnad American Scientist Open Access Forum Preprints, Postprints, Peer Review, and Institutional vs. Central Self-ArchivingComments on: Ginsparg, Paul (2006) As We May Read. The Journal of Neuroscience, September 20, 2006, 26(38): 9606-9608 doi:10.1523/JNEUROSCI.3161-06.2006Arxiv is a Central Repository (CR) in which physicists (mostly, and many mathematicians, and some computer scientists) have been self-archiving their unrefereed preprints and their peer-reviewed postprints since 1991. It is important to keep in mind that researchers self-archive preprints as well as postprints, because it makes a big difference whether one extrapolates from Arxiv as a preprint CR or a postprint CR, as we shall see below. It is also pertinent to bear in mind that Arxiv is indeed a Central Repository (CR), because there is now a growing movement toward distributed Institutional Repositories (IRs). The IR movement was facilitated by the Open Archives Initiative (OAI) Protocol for Metadata Harvesting, which renders all IRs and CRs interoperable: the OAI Protocol was in turn created partly as a result of an initiative from Arxiv. As a consequence of the OAI Protocol, all OAI-compliant IRs and CRs are interoperable: their metadata can be harvested into search engines that treat all of their contents as if they were in one big virtual CR. "As a pure dissemination system, [Arxiv] operates at a factor of 100-1000 times lower [1.0% - 0.1%] in cost than a conventionally peer-reviewed system (Ginsparg, 2001)."This is true, but it is tantamount to saying that as a pure dissemination system, photocopying the articles published in journals operates at a fraction of the cost of publishing a journal: A fraction, but a parasitic fraction, for without the journal, there would be nothing to either photocopy or distribute in Arxiv. Nothing but the unrefereed preprint, that is. And this brings us face to face with the fundamental question: What are the true costs of peer review, and peer review alone? The peers (scarce, overused resource though they are) review for free, so it is not their services whose costs we are talking about, but the cost of implementing the peer review: processing the submissions, picking the referees, processing their reports, deciding what revisions need to be done to meet the journal's quality standards for acceptance, and deciding -- perhaps again by consulting the referees -- whether those revisions have been successfully done. The selection of referees and the decision as to what needs to be done is usually made by a qualified, answerable super-peer: the editor (or a board of editors). The editor(s) services, and the clerical services for processing submissions, communicating with referees, and processing referee reports are the costs involved -- and these include not just accepted papers, but rejected ones too (with some journals' rejection rates being over 90%). In other words, peer-reviewed journal publishing is not a "pure dissemination system." Implementing the peer review costs some money too. There are estimates of what it costs (about $500 per paper was the average estimate a few years ago, which is between one-third and one-sixth of the charge per article that today's "Open Choice" journals are currently proposing -- although a few journals with high rejection rates have suggested a figure of $10,000 per article, without making it clear whether this represents their costs per article or their income per article). The annual cost per paper in Arxiv, to Arxiv, has been estimated at about $10 (a few years ago), so this is indeed somewhere between 2% of the low-end estimate and 0.1% of the high-end estimate. If we include the cost of keying in the deposit to the depositor, it's a few pennies more. But what do these figures mean? Why compare the cost of online dissemination alone with the cost of peer review (or any of the other values a journal adds, such as the print edition, copy-editing, reference-checking, and mark-up)? "with many of the production tasks automatable or off-loadable to the authors, the editorial costs will then dominate the costs of an unreviewed distribution system by many orders of magnitude."Translation: Online dissemination of unrefereed preprints alone costs a lot less than peer-reviewed publication. True, but what follows from that? Peer-reviewed publication costs a lot more than photo-copying too, but what authors photocopy and distribute is their peer-reviewed publications, not just their unrefereed preprints. "Although the most recently submitted articles have not yet necessarily undergone formal review, the vast majority of the articles can, would, or do eventually satisfy editorial requirements somewhere.... [Arxiv's moderated] submissions are at least 'of refereeable quality'."Every paper is first an unrefereed preprint -- and then, eventually, most are revised into peer-reviewed, accepted articles (postprints). Hence if preprints are deposited in Arxiv at all, it stands to reason that Arxiv's most recently deposited (sic) papers (sic) have not yet undergone peer review. Tune in a year later, and they will have been, with the revised postprint now also deposited. Preprints and postprints are deposited rather than "submitted" to IRs or CRs, because an archive is merely a repository, not a certifier of having met a peer-reviewed journal's quality standards: let's reserve "submission" for the attempt to meet a journal's peer-review quality standards. Moreover, unrefereed preprints are merely papers, not articles; they become articles when they have been accepted for publication by a peer-reviewed journal. This is not pedantry or formalism. It is merely the sorting out of what has and has not met known quality control standards. The tag certifying this is currently the journal name, with its established quality level and track-record. A peer-reviewed journal (apart from its function as an access-provider) is a peer-review service-provider/certifier, publicly answerable for its quality standards with its own prestige and reputation. And authors are in turn answerable to the editor and referees, for meeting their standards for acceptance; revision is not optional but obligatory, a condition on acceptance for publication. Hence earning the tag certifying acceptance is a dynamic, interactive process, and not merely a pass/fail system. Publication is even less like a pass/fail system in that in most fields there is a hierarchy of journals, with a range of peer-review standards, from the one or few most rigorous ones at the top (usually the ones with the highest rejection rates), all the way down to what is sometimes almost a vanity press at the bottom (little better than an unrefereed preprint). These differences in quality standards are known and relied upon in the field. And papers are not really published or unpublished: Most are published, eventually, but at their own quality level. The journals are all autonomous, independent of the authors and the authors' institutions, each dependent on its own established standards for quality and selectivity. Users are in turn dependent on each journal's public track record in deciding what to trust. It is not at all clear what an IR's or CR's certification of which of its deposits is "of refereeable quality" might mean to busy researchers who need to know whether a paper is worth risking their limited time to read and try to use, apply and build upon. Users currently do this by seeing whether and where it has been published (with the journal name and track record serving as their indicator of the article's probable level of quality, reliability and validity). Unrefereed preprints have always been something handled with care, having only the author's name, institution and prior track-record as a guide to their reliability. Is Arxiv's tag of being "of refereeable quality" meant to serve as a further guide? or as a substitute for something? "[P]roposed modifications of the peer review include a two-tier system (for more details, see Ginsparg, 2002), in which, on a first pass, only some cursory examination or other pro forma certification is given for acceptance into a standard tier. At some later point, a much smaller set of articles would be selected for more extensive evaluation."This is a speculative hypothesis. It is no doubt being tested to see whether it works, whether it delivers results of quality and useability comparable to standard peer review, whether it is cost-effective, and whether it can replace journals. But as it stands, the hypothesis alone does not tell us whether and how well it will work; Arxiv is certainly not evidence for the validity of this hypothesis, since virtually all papers in Arxiv still undergo standard peer review. Arxiv is merely a CR that provides Open Access (OA) to both the preprints and the postprints. "using standard search engines, more than one-third of the high-impact journal articles in a sample of biological/medical journals published in 2003 were found at nonjournal Web sites (Wren, 2005)."This is very interesting. This is the higher end of a self-archiving rate that we have found to range between about 5% and 25% across disciplines. Physics is of course even higher (mostly because of Arxiv) and computer science higher still (see Citeseer, a google-style harvester of distributed locally deposited papers). "at least 75% of the publications listed [in neuroscience] were freely available either via direct links from the above Web page or via a straightforward Web search for the article title."This is even more interesting. It means that in such fields the majority of the articles -- note that we are almost certainly not talking about unrefereed preprints here but about peer-reviewed postprints -- are being self-archived already, so the only thing that remains to be done is to deposit (or harvest) them into the author's own OAI-compliant IR rather than a random website, to maximise visibility, harvestability, and impact. "The enormously powerful sorts of data mining and number crunching that are already taken for granted as applied to the open-access genomics databases can be applied to the full text"Indeed. And semantic and scientometric analyses too (though article texts are not quite the same thing as the research data on which the articles are based, hence the analogy with the genomics data base may be a bit misleading). "it is likely that more research communities will join some form of global unified archive system without the current partitioning and access restrictions familiar from the paper medium"What makes it most likely is the self-archiving mandates proposed or already adopted the world over (e.g., RCUK, Wellcome Trust, FRPAA, EC, plus individual institutional self-archiving mandates: CERN, Southampton, QUT, Minho). But the deposits will not be done in one global CR, nor in a CR like Arxiv for each discipline or combination of disciplines. With the advent of the OAI protocol, all IRs and CRs are interoperable, and since the research institutions themselves are the primary research providers, with the direct interest inshowcasing their own research output as well as maximizing its uptake, usageand impact, the natural place for them to deposit their own output is in their own IRs. Any central collections can be gathered via OAI harvesting. Institutions are also best placed to monitor and reward compliance with self-archiving mandates, both their own institutional mandates and those of the funders of their institutional research output. Arxiv has played an important role in getting us where we are, but it is likely that the era of CRs is coming to a close, and the era of distributed, interoperable IRs is now coming into its own in an entirely natural way, in keeping with the distributed nature of the Net/Web itself. Stevan Harnad American Scientist Open Access Forum Wednesday, October 4. 2006France's HAL, OAI interoperability, and Central vs Institutional RepositoriesOn Mon, 2 Oct 2006, Franck Laloë [FL] wrote: FL: "Hal (and I presume Prodinra [INRA]) are of course OAI-PMH compatible, and of course can be harvested within this protocol and others. This compatibility is a necessary condition for an archive to be useful to the scientific community. But a necessary condition is not always sufficient. We need more interoperability than just that possible within OAI-PMH; Hal meets this requirement. -- I know that Stevan and others will disagree with the last sentence above..."But I don't disagree at all! The more interoperability the better! What I am still very keen to know is the following: The most important question is (3), because what is becoming increasingly clear is that -- be they as interoperable as one could possibly wish -- near-empty repositories are not very useful! The spontaneous (unmandated) self-archiving rate worldwide is about 15% (with a few disciplines, e.g. physics, well above 15%, but most disciplines at or below 15%), whereas the mandated deposit rate climbs toward 100% within a few years of adoption (as predicted by Alma Swan's international surveys, and confirmed by Arthur Sale's analyses comparing mandated and unmandated deposit rates).(1) How is it proposed to get all of France's research output into HAL? So where do things stand in France regarding self-archiving mandates (whether for local institutional deposit or national deposit in HAL)? FL: "There are several reasons ["why... deposits need to be directly in HAL, rather than in each author's own Institutional Repository (IR), then harvested by HAL"]:But I don't understand: Can these complex affiliation metadata not be provided (rather trivially)? FL: "This structure of metadata in Hal is richer than OAI-PMH (in its present state)."I understand: HAL does richer tagging than the OAI-PMH IRs. But that tagging has to be added in any case: So why not add to the metadata harvested from the distributed deposits in each author's own IR? FL: "If we harvested all OAI-PMH compliant archives, this would introduce redundancy"So what? And there are ways to either remove or coalesce duplicates. (The problem today -- we cannot remind ourselves often enough -- is too few deposits [15%], not too many, nor too many duplicates!) FL: "documents with incomplete metadata (most institutional repositories only mention their institution, not the contribution of others, etc..)."We've agreed that if HAL aspires to have richer metadata than IRs with their OAI-PMH then the extra tags have to be added; this is not an answer to the question of why HAL insists on direct deposits, rather than harvesting from IRs. And you already mentioned earlier the baroque intricacies of institutional affiliation -- but not why you don't think this trivial problem cannot be handled easily by software, with the help of IR affiliations, lists of co-authors and (why not) central authoritative lists chronicling all French authors' current (and past!) complexities of affiliation, turning them into explicit metadata tags... FL: "Many OAI repositories do not guarantee sufficient quality, and even access to the full text."The 1st, 2nd and Nth immediate problem today is lack of content, not low-quality metadata: The texts [85% of them] are not deposited at all. The OA movement, and OA self-archiving mandates, are endeavouring to get that content deposited. Authors' own IRs are the natural place to deposit it, and to mandate depositing it. Then (as agreed above) HAL can, if it wishes, harvest that content, and improve its metadata. (Again, this is no argument against harvesting, or in favour of direct deposit in HAL.) As to texts that are deposited in IRs but not made OA: I wish that were the only remaining problem, for I guarantee that if it were, it would solve itself in short order. (The OA metadata would elicit email eprint requests, and authors would soon tire of emailing eprints and would instead set access to their deposited full-texts as OA instead of Closed Access.) But we do not have all or most or even much of the target literature -- the peer-reviewed research corpus -- deposited in IRs in Closed Access, with only their (low-quality) metadata accessible: At least 85% percent of OA's target content is not deposited at all. So it seems to me HAL would benefit as much as everyone else from a self-archiving mandate that would get all that content deposited; so the only question is who will mandate it to be deposited, and where? So far, the two natural candidate mandaters are the researchers' own institutions and funders. Clearly institutions have an interest in mandating that the deposit should be in their own IRs (for institutional visibility, prestige, and record-keeping). Funders (although some unthinkingly insist on central deposits today, e.g., in PubMed Central) are mostly indifferent to where their funded research is deposited, as long as it is OAI-compliant and OA. So many mandate depositing in the researcher's own IR too. And PubMed Central should be asking itself the same questions I am asking you about HAL: Why not deposit in each researcher's own OAI-compliant IR and simply harvest from there? Institutions have a direct institutional interest in their own IRs; they are the ones that can best monitor and reward compliance with self-archiving mandates; and the spectrum of disciplines at research institutions (mostly universities) effectively cover all of OA's target content space (whereas central disciplinary and multidisciplinary repositories do not). A national repository like HAL is a very good idea, but unless the problem of the means of mandating and monitoring direct self-archiving in HAL by all French researchers has an immediate solution, at the very least a hybrid deposit system would seem to be optimal: either researchers deposit in their own IRs (subsequently harvested and enhanced by HAL) or directly in HAL; but deposit they must. FL: "Hal includes certification procedures (which we call "stamps") which do not exist in other open archives."That's fine, but the non-existence that is the immediate problem is not certificates but deposits! At least 85% of French research output is not being self-archived at all. Institutional and funder self-archiving mandates can remedy this, but are all or most of France's research institutions more likely to agree to mandate and monitor depositing all of their own output in HAL, or in their own IRs? The "stamps" could come either way, either via harvesting from IRs or via direct deposits. FL: "In brief, we want Hal to be an homogeneous system, really usable by the reader, and by labs (even if they belong to several institutions) and institutions - all this through a single entry into the system."If HAL can become a direct entry point for all French research institutions, and they all agree to a means of mandating and monitoring compliance, nolo contendere! But what is sure is that a central repository and central depositing is not the only way to get an OA corpus usable by all (authors and users), in France and worldwide. On the contrary, the nature of the Internet, the Web, the OAI protocol and any other richer metadata tagging schemes is such that distributed interoperability -- rather than a central locus and central management -- is far more likely to prove to be the successful means of generating and using the OA corpus, in France and worldwide. FL: "For instance, my lab belongs to 4 institutions, we do not want to put our articles into four open archives; one is enough."First, if the 4 institutions don't want or need their research output to be deposited in their own IRs, there is no need to do it. Perhaps the lab itself will want to have its own IR. Moreover, harvesting works N ways: Once a paper and its (OAI) metadata are deposited in one IR, other IRs (as well as Central Repositories like HAL or PubMed Central or Arxiv) can harvest it; or the author can import/export it to his multiple IR affiliations. (And let us not forget that even direct deposit takes less than ten minutes worth of keystrokes!) In other words, with OAI harvestability, yes, one deposit is enough. FL: "I am just explaining what we do, and the strategy we chose (after much discussion!). I am not claiming that it is the best in the world, or even superior to others; actually, I know that you do not approve it, Stevan. But I personally believe in it, because I feel that it meets the quality that is necessary to build a real tool for research."Franck, it has nothing to do with approval or disapproval. Whatever system results in 100% of French research output being made OA (soon!) -- whether by mandating direct deposit in HAL, or mandating local IR mandating, or even (mirabile dictu) by having all journals convert to OA publishing -- realises the goal of the OA movement: 100% OA for peer-reviewed research output, now. But is HAL's policy of central deposit and metadata enhancement sufficient to generate that 100% self-archiving? For if not, then whatever other desiderata it may be providing, it is not providing OA's target content. FL: "one can easily extract a local institutional repository from Hal, and even import all the data locally, if useful."I don't doubt it. But you have not yet told me how you propose to get all that content deposited in HAL in the first place, so that institutions can then harvest back their own content from it: On the face of it, it would seem that the institutions should be depositing their own content in their own IRs directly, and HAL should be harvesting it, not vice versa. But if you do have a plan for a national mandate to deposit directly in HAL, I would say all this discussion is moot. Without such a plan, however, this discussion is beside the point (at least insofar as OA is concerned). FL: "one can also transfer documents to Hal from local systems using the so called "webservice" techniques. In other words you can load documents into Hal from your local system for electronic documents, without knowing anything about Hal, provided that your metadata are Hal compatible. This is what several institutions are now doing in France."The French institutions that have already succeeded in getting their research output into their own IRs -- whether merely OAI-compliant IRs or HAL-compliant IRs -- have already succeeded in solving the problem we (or at least I!) am discussing here, for whatever contents they have succeeded in getting deposited. My guess is that if these deposits are unmandated, than they represent about 15% of those institutions' annual research output, and we are back where we started. The issue, au fond, is not where papers are deposited, but whether they are deposited. The only reason I keep harping on institutional IR depositing rather than central depositing is that institutions are the primary content providers, in all research disciplines, and hence their own IRs are the natural place to require their own researchers to self-archiving their own research output. Moreover, institutions cover all research disciplines, hence all of OA's target content space. It is virtually certain that the only way to attain 100% OA self-archiving is via self-archiving mandates from researchers' institutions and funders. Hence the only real question about IR deposit vs. HAL deposit in France is whether the probability of a successful pandisciplinary, paninstitutional national mandate to deposit in HAL is greater in France than the probability of institutional and funder mandates to self-archive institutionally. What is best for France is whichever of these is in fact more likely. FL: "Let me finally add that Hal has been conceived to combine the advantage of disciplinary open archives (what scientists want)."I think that what you wanted to say, Franck, was that (many) scientists want to be able to search and access all and only the relevant research in their own disciplines. That they want it all to be in one discipline-based "archive," and that that archive must have been deposited in directly rather than harvested -- and even that the realisation of these wishes requires the full richness of HAL's proposed metadata -- is rather a theoretical assumption on the part of some, rather than an objective statement of "what scientists want"... FL: "and institutional archives (which are indispensable if we want institutions to push scientists to deposit their [research output]). "This, I think, is closer to assumption-free objectivity: Institutions do want their own output in their own IRs and not just in some external discipline-based collective database. But here I would agree with you: Harvesting could work in either direction, to give everyone what they want. But harvesting will not get undeposited content deposited; only mandates will. So the question is whether institutions (and funders) are more likely to be pushed to push their researchers to deposit their research output in (1) international disciplinary archives like Arxiv or PubMed Central, (2) national omnibus archives like HAL, or (3) their own institutional IRs? FL: "You can create portals of Hal that are institutional, with the logo, words, etc.. of the institution, for both upload and download."I agree completely that harvesting can go either way, so if, mirabile dictu, HAL succeeded in getting all or most of French research output in all disciplines directly deposited in HAL, then it would be trivial to generate virtual IRs for each institution via back-harvesting. But how do you propose to get the content deposited in HAL in the first place? You seem to be focussed on centrality and metadata enrichment, but we need to hear about how you plan to get the content (and how much of it): The target is 100% of French research output. The baseline today is 15% spontaneous self-archiving: How do you plan to get from 15% to 100%, and when? FL: "But at the same time all the documents go to the same data base. This is technically possible, but requires the solid structure of metadata that I described above."It requires something much harder to get the solid metadata structure: it requires 100% of the target content! FL: "I hope that I have explained the situation clearly."As Fermat (or the hopeful builder of the perpetuum mobile) would have conceded: there are still a few little details missing. In this case, the detail concerns how you plan to get HAL filled. For without that, we are talking about raising the quality standards and price for a product that does not yet have any customers (apart from the 15% spontaneous baseline)... Stevan Harnad American Scientist Open Access Forum
« previous page
(Page 100 of 113, totaling 1129 entries)
» next page
|
QuicksearchSyndicate This BlogMaterials You Are Invited To Use To Promote OA Self-Archiving:
Videos:
The American Scientist Open Access Forum has been chronicling and often directing the course of progress in providing Open Access to Universities' Peer-Reviewed Research Articles since its inception in the US in 1998 by the American Scientist, published by the Sigma Xi Society. The Forum is largely for policy-makers at universities, research institutions and research funding agencies worldwide who are interested in institutional Open Acess Provision policy. (It is not a general discussion group for serials, pricing or publishing issues: it is specifically focussed on institutional Open Acess policy.)
You can sign on to the Forum here.
ArchivesCalendar
CategoriesBlog AdministrationStatisticsLast entry: 2018-09-14 13:27
1129 entries written
238 comments have been made
Top ReferrersSyndicate This Blog |