Perils of Press-Release Journalism: NSF, U. Chicago, and Chronicle of Higher Education

---

Update Jan 1, 2010: See Gargouri, Y; C Hajjem, V Larivière, Y Gingras, L Carr,T Brody & S Harnad (2010) “Open Access, Whether Self-Selected or Mandated, Increases Citation Impact, Especially for Higher Quality Research”
Update Feb 8, 2010: See also "Open Access: Self-Selected, Mandated & Random; Answers & Questions"

---

In response to my critique of his Chronicle of Higher Education posting on Evans and Reimer's (2009) Science article (which I likewise critiqued, though much more mildly), I got an email from Paul Basken asking me to explain what, if anything, he had got wrong, since his posting was based entirely on a press release from NSF (which turns out to be a relay of a press release from the University of Chicago, E & R's home institution). Sure enough, the silly spin originated from the NSF/Chicago Press release (though the buck stops with E & R's own vague and somewhat tendentious description and interpretation of some of their findings). Here is the NSF/Chicago Press Release, enhanced with my comments, for your delectation and verdict:

---

NSF/U.CHICAGO:
"If you offer something of value to people for free while someone else charges a hefty sum of money for the same type of product, one would logically assume that most people would choose the free option. According to new research in today's edition of the journal Science, if the product in question is access to scholarly papers and research, that logic might just be wrong. These findings provide new insight into the nature of scholarly discourse and the future of the open source publication movement [sic, emphasis added]."

(1) If you offer something valuable for free, people will choose the free option unless they've already paid for the paid option (especially if they needed -- and could afford -- it earlier).

(2) Free access after an embargo of a year or more is not the same "something" as immediate free access. Its "value" for a potential user is lower. (That's one of the reasons institutions keep paying for subscription/license access to journals.)

(3) Hence it is not in the least surprising that immediate (paid) print-on-paper access + online access (IP + IO) generates more citations than immediate (paid) print-on-paper access (IP) alone.

(4) Nor is it surprising that immediate (paid) print-on-paper access + online access + delayed free online access (IP +IO + DF) generates more citations than just immediate (paid) print-on-paper + online access (IP + IO) alone -- even if the free access is provided a year or longer after the paid access.

(5) Why on earth would anyone conclude that the fact that the increase in citations from IP to IP + IO is 12% and the increase in citations from IP + IO to IP + IO + DF is a further 8% implies anything whatsoever about people's preference for paid access over free access? Especially when the free access is not even immediate (IF) but delayed (DF) and the 8% is an underestimate based on averaging in ancient articles: see E & R's supplemental Figure S1(c), right [with thanks to Mike Eisen for spotting this one!].

NSF/U.CHICAGO:
"Most research is published in scientific journals and reviews, and subscriptions to these outlets have traditionally cost money--in some cases a great deal of money. Publishers must cover the costs of producing peer-reviewed publications and in most cases also try to turn a profit. To access these publications, other scholars and researchers must either be able to afford subscriptions or work at institutions that can provide access.

"In recent years, as the Internet has helped lower the cost of publishing, more and more scientists have begun publishing their research in open source [sic] outlets online. Since these publications are free to anyone with an Internet connection, the belief has been that more interested readers will find them and potentially cite them. Earlier studies had postulated that being in an open source [sic] format could more than double the number of times a journal article is used by other researchers."

What on earth is an "open source outlet"? ("Open source" is a software matter.) Let's assume what's meant is "open access"; but then is this referring to (i) publishing in an open access journal, to (ii) publishing in a subscription journal but also self-archiving the published article to make it open access, or to (iii) self-archiving an unpublished paper?

What (many) previous studies had measured (not "postulated") was that authors (ii) publishing in a subscription journal (IP + IO) and also self-archiving their published article to make it Open Access (IP + IO + OA) could more than double their citations, compared to IP + IO alone.

NSF/U.CHICAGO:
"To test this theory, James A. Evans, an assistant professor of sociology at the University of Chicago, and Jacob Reimer, a student of neurobiology also at the University of Chicago, analyzed millions of articles available online, including those from open source publications [sic] and those that required payment to access."

No, Evans & Reimer (E & R) did nothing of the sort; and no "theory" was tested (nor was there any theory).

E & R only analyzed articles from subscription access journals before and after the journals made them accessible online (to paid subscribers only) (i.e., IP vs IP + IO) as well as before and after the journals made the online version accessible free for all (after a paid-access-only embargo of up to a year or more: i.e., IP +IO vs IP + IO + DF). E & R's methodology was based on comparing citation counts for articles within the same journals before and after being made free online (by the journal) following delays of various lengths.

NSF/U.CHICAGO:
"The results were surprising. On average, when a given publication was made available online after being in print for a year, being published in an open source format [sic] increased the use of that article by about 8 percent. When articles are made available online in a commercial format a year after publication, however, usage increases by about 12 percent."

In other words, the citation count increase from just (paid) IP to (paid) IP + IO was 12% and the citation count increase from just (paid) IP + IO to (paid) IP + IO + DF was a further 8%. Not in the least surprising: Making paid-access articles accessible online increases their citations, and making them free online (even if only after a delay of a year or longer) increases their citations still more.

What is surprising is the rather absurd spin that this press release appears to be trying to put on this decidedly unsurprising finding.

NSF/U.CHICAGO:
"'Across the scientific community,' Evans said in an interview, 'it turns out that open access does have a positive impact on the attention that's given to the journal articles, but it's a small impact.'"

We already knew that OA increased citations, as the many prior published studies have shown.  Most of those studies, however, were based on immediate OA (i.e., IF), not embargoed OA. What E & R do show, interestingly, is that even delaying OA for a year or more still increases citations, though (unsurprisingly) not as much as immediate OA (IF) does.

NSF/U.CHICAGO:
"Yet Evans and Reimer's research also points to one very positive impact of the open source movement [sic] that is sometimes overlooked in the debate about scholarly publications. Researchers in the developing world, where research funding and libraries are not as robust as they are in wealthier countries, were far more likely to read and cite open source articles."

A large portion of the citation increase from (delayed) OA turns out to come from Developing Countries (refuting Frandsen's recent report to the contrary). This is a new and useful finding (though hardly a surprising one, if one does the arithmetic). (A similar analysis, within the US, comparing citations from America's own "Have-Not" Universities (with the smaller journal subscription budgets) with its Harvards might well reveal the same effect closer to home, though probably at a smaller scale.)

NSF/U.CHICAGO:
"The University of Chicago team concludes that outside the developed world, the open source movement [sic] 'widens the global circle of those who can participate in science and benefit from it.'"

And it will be interesting to test for the same effect comparing the Harvards and the Have-Nots in the US -- but a more realistic estimate might come from looking at immediate OA (IF) rather than just embargoed OA (DF).

NSF/U.CHICAGO:
"So while some scientists and scholars may chose to pay for scientific publications even when free publications are available, their colleagues in other parts of the world may find that going with open source works [sic] is the only choice they have."

It would be interesting to hear the authors of this NSF/Chicago press release -- or E & R, for that matter -- explain how this paradoxical "preference" for paid access over free access was tested during the access embargo period...

Stevan Harnad
American Scientist Open Access Forum