Where There's No Access Problem There's No Open Access Advantage

---

Update Jan 1, 2010: See Gargouri, Y; C Hajjem, V Larivière, Y Gingras, L Carr,T Brody & S Harnad (2010) “Open Access, Whether Self-Selected or Mandated, Increases Citation Impact, Especially for Higher Quality Research”
Update Feb 8, 2010: See also "Open Access: Self-Selected, Mandated & Random; Answers & Questions"

---

---

SUMMARY: Kurtz & Henneken (2007) report that the citation advantage of astrophysics papers self-archived as preprints in Arxiv is caused by (1) Early Advantage (EA) (earlier citations for papers self-archived earlier) and (2) Quality Bias (QB) (a self-selection bias toward self-archiving higher quality papers) and not by (3) Open Access (OA) itself (being freely accessible online to those who cannot afford subscription-toll access).
   K & H suggest: "[T]here is no 'Open Access Advantage' for papers from the Astrophysical Journal" because "in a well funded field like astrophysics essentially everyone who is in a position to write research articles has full access to the literature."
   This seems like a perfectly reasonable explanation for K&H's findings. Where there is no access problem, OA cannot be the cause of whatever higher citation count is observed for self-archived articles.
   We (Hajjem & Harnad 2007) have conducted a similar study, but across the full spectrum of disciplines, measuring the citation advantage for mandated and unmandated self-archiving for articles from four Institutional Repositories that have self-archiving mandates, each compared to articles in the very same journal and year by authors from other institutions (on the assumption that mandated self-archiving should have less of a self-selection Quality Bias than unmandated self-archiving).
   We again confirmed the citation advantage for self-archiving, and found no difference in the size of that advantage for mandated and unmandated self-archiving. It is likely that the size of the access problem differs from field to field, and with it the size of the OA citation advantage. Evidence suggests that most fields are not nearly as well-heeled as astrophysics.

---

Kurtz & Henneken (2007) report a very interesting new result:

"We demonstrate conclusively that there is no 'Open Access Advantage' for papers from the Astrophysical Journal. The two to one citation advantage enjoyed by papers deposited in the arXiv e-print server is due entirely to the nature and timing of the deposited papers. This may have implications for other disciplines."

Earlier, Kurtz et al. (2005) had shown that the lion's share of the citation advantage of astrophysics papers self-archived as preprints in Arxiv was caused by (1) Early Advantage (EA) (earlier citations for papers self-archived earlier) and (2) Quality Bias (QB) (a self-selection bias toward self-archiving higher quality papers) and not by (3) Open Access (OA) itself (being freely accessible online to those who cannot afford subscription-toll access).

Kurtz et al. explained their finding by suggesting that:

"in a well funded field like astrophysics essentially everyone who is in a position to write research articles has full access to the literature."

This seems like a perfectly reasonable explanation for their findings. Where there is no access problem, OA cannot be the cause of whatever higher citation count is observed for self-archived articles. Moed (2007) has recently reported a similar result in Condensed Matter Physics, and so have Davis & Fromerth (2007) in 4 mathematics journals.

Kurtz & Henneken's latest study confirms and strengthens their prior finding: They compared citation counts for articles published in two successive years of the Astrophysical Journal. For one of the years, the journal was freely accessible to everyone; for the other it was only accessible to subscribers. The citation counts for the self-archived articles, as expected, were twice as high as for the non-self-archived articles. They then compared the citation-counts for non-self-archived articles in the free-access year and in the toll-access year, and found no difference. They concluded, again, that OA does not cause increased citations.

But of course K&H's prior explanation -- which is that there is no access problem in astrophysics -- applies here too: It means that in a field where there is no access problem, whatever citation advantage comes from making an article OA by self-archiving cannot be an OA effect.

K&H conclude that "[t]his may have implications for other disciplines."

It should be evident, however, that the degree to which this has implications for other disciplines depends largely on the degree to which it is true in other disciplines that "essentially everyone who is in a position to write research articles has full access to the literature."

We (Hajjem & Harnad 2007) have conducted (and are currently replicating) a similar study, but across the full spectrum of disciplines, measuring the citation advantage for mandated and unmandated self-archiving for articles from 4 Institutional Repositories that have self-archiving mandates (three universities plus CERN), each compared to articles in the very same journal and year by authors from other institutions (on the assumption that mandated self-archiving should have less of a self-selection quality bias than unmandated self-archiving).

Figure 1. Self-Selected Self-Archiving vs. Mandated Self-Archiving: Within-Journal Citation Ratios (for 2004, 4 mandating institutions, all fields). S = citation counts for articles self-archived at institutions with (Sm) and without (Sn) a self-archiving mandate. N = citation counts for non-archived articles at institutions with (Nm) and without (Nn) mandate (i.e., Nm = articles not yet compliant with mandate). Grand average of (log) S/N ratios (106,203 articles; 279 journals) is the OA advantage (18%); this is about the same as for Sn/Nn (27972 articles, 48 journals, 18%) and Sn/N (17%); ratio is higher for Sm/N (34%), higher still for Sm/Nm (57%, 541 articles, 20 journals); and Sm/Sn = 27%, so self-selected self-archiving does not yield more citations than mandated (if anything, it is rather the reverse). (All six within-pair differences are significant: correlated sample t-tests.)
(NB: preliminary, unrefereed results.)

We again confirmed the citation advantage for self-archiving, and found no difference in the size of that advantage for mandated and unmandated self-archiving. (The finding of an equally large self-archiving advantage for mandated and unmandated self-archiving was also confirmed for CERN, whose articles are all in physics -- although one could perhaps argue that CERN articles enjoy a quality advantage over articles from other institutions.)

A few closing points:

(1) It is likely that the size of the access problem differs from field to field, and with it the size of the OA citation advantage. Evidence suggests that most fields are not nearly as well-heeled as astrophysics. According to a JISC survey, 48% of researchers overall (biomedical sciences 53%, physical/engineering sciences 42%, social sciences 47%, language/linguistics 48% and arts/humanities 53%) have difficulty in gaining access to the resources they need to do their research. (The ARL statistics on US university serials holdings is consistent with this.) The overall access difficulty is roughly congruent with the reported OA access advantage.

(2) In Condensed Matter physics (and perhaps also mathematics) there is the further complication that it is mostly just preprints that are being self-archived, which reduces the scope for observing any postprint advantage, as opposed to just an Early Advantage (EA). (In fields that do have a significant access problem, the "Early Advantage" for postprints is the OA Advantage!) OA self-archiving mandates (and the OA movement in general) target refereed, accepted postprints, not unrefereed preprints.

(3) The positive correlation is between citation counts and the number of self-archived articles in each citation-range (Hajjem et al 2005). This could be caused by Quality Bias (QB) (higher quality articles being more likely to be self-archived) or Quality Advantage (QA) (higher quality articles benefiting more from being self-archived) or both. (The top 50% of articles tend to be cited 10 times as often as the bottom 50% [Seglen 1992]. This has nothing to do with OA or self-archiving.)

(4) Citations are of course not the only potential advantage of self-archiving. Downloads can increase too (Brody et al. 2006).

(5) We are now in the Institutional Repository (IR) era. Arxiv is a venerable Central Repository, one of the biggest and oldest and most successful of them all. But, institutions are the sources of OA content, institutions are in the best position to mandate self-archiving, and institutions share with their authors the benefits of enhancing the accessibility, usage and impact of their research output. Moreover, the nature of the web is distributed local websites, harvested by central service providers, rather than central websites.

Stevan Harnad

Brody, T., Harnad, S. and Carr, L. (2006) Earlier Web Usage Statistics as Predictors of Later Citation Impact. Journal of the American Association for Information Science and Technology (JASIST) 57(8) pp. 1060-1072.

Davis, P. M. and Fromerth, M. J. (2007) Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles? Scientometics, Vol. 71, No. 2.

Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year Cross-Disciplinary Comparison of the Growth of Open Access and How it Increases Research Citation Impact. IEEE Data Engineering Bulletin 28(4) pp. 39-47.

Hajjem, C. and Harnad, S. (2007) The Open Access Citation Advantage: Quality Advantage Or Quality Bias? Technical Report, Electronics and Computer Science, University of Southampton.

Kurtz, M. J. and Henneken, E. A. (2007) Open Access does not increase citations for research articles from The Astrophysical Journal. Preprint deposited in arXiv September 6, 2007.

Kurtz, M. J., Eichhorn, G., Accomazzi, A., Grant, C. S., Demleitner, M., Murray, S. S. (2005, The Effect of Use and Access on Citations. Information Processing and Management, 41, 1395-1402)

Moed, H. F. (2007) The effect of 'open access' on citation impact: An analysis of ArXiv's condensed matter section, Journal of the American Society for Information Science and Technology (JASIST) , August 30, 2007.

Seglen, P. O. (1992) The skewness of science. Journal of the American Society for Information Science 43:628-38