Open Access Archivangelism

The Times Higher Education Supplement (THES) has reported the results of a study they commissioned by Evidence Ltd that found that the ranking criteria for assessing and rewarding research performance in the UK Research Assessment Exercise (RAE) changed from RAE 2001 to RAE 2008. The result is that citations, which correlated highly with RAE 2001, correlated less highly with RAE 2008, so a number of universities whose citation counts had decreased were rewarded more in 2008, and a number of universities whose citation counts had increased were rewarded less.

(1) Citation counts are only one (though an important one) among many potential metrics of research performance.

(2) If the RAE peer panel raters' criteria for ranking the universities varied or were inconsistent between RAE 2001 and RAE 2008 then that is a problem with peer ratings rather than with metrics (which, being objective, remain consistent).

(3) Despite the variability and inconsistency, peer ratings are the only way to initialise the weights on metrics: Metrics first have to be jointly validated against expert peer evaluation by measuring their correlation with the peer rankings, discipline by discipline; then the metrics' respective weights can be updated and fine-tuned, discipline by discipline, in conjunction with expert judgment of the resulting rankings and continuing research activity.

(4) If only one metric (e.g., citation) is used, there is the risk that expert ratings will simply echo it. But if a rich and diverse battery of multiple metrics is jointly validated and initialized against the RAE 2008 expert ratings, then this will create an assessment-assistant tool whose initial weights can be calibrated and used in an exploratory way to generate different rankings, to be compared by the peer panels with previous rankings as well as with new, evolving criteria of research productivity, uptake, importance, influence, excellence and impact.

(5) The dawning era of Open Access (free web access) to peer-reviewed research is providing a wealth of new metrics to be included, tested and assigned initial weights in the joint battery of metrics. These include download counts, citation and download growth and decay rates, hub and authority scores, interdisciplinarity scores, co-citations, tag counts, comment counts, link counts, data-usage, and many other openly accessible and measurable properties of the growth of knowledge in our evolving "Cognitive Commons."

Brody, T., Kampa, S., Harnad, S., Carr, L. and Hitchcock, S. (2003) Digitometric Services for Open Archives Environments. In Proceedings of European Conference on Digital Libraries 2003, pp. 207-220, Trondheim, Norway.

Brody, T., Carr, L., Harnad, S. and Swan, A. (2007) Time to Convert to Metrics. Research Fortnight pp. 17-18.

Brody, T., Carr, L., Gingras, Y., Hajjem, C., Harnad, S. and Swan, A. (2007) Incentivizing the Open Access Research Web: Publication-Archiving, Data-Archiving and Scientometrics. CTWatch Quarterly 3(3).

Carr, L., Hitchcock, S., Oppenheim, C., McDonald, J. W., Champion, T. and Harnad, S. (2006) Extending journal-based research impact assessment to book-based disciplines. Technical Report, ECS, University of Southampton.

Hajjem, C., Harnad, S. and Gingras, Y. (2005) Ten-Year Cross-Disciplinary Comparison of the Growth of Open Access and How it Increases Research Citation Impact. IEEE Data Engineering Bulletin 28(4) pp. 39-47.

Harnad, S. (2001) Research access, impact and assessment. Times Higher Education Supplement 1487: p. 16.

Harnad, S. (2007) Open Access Scientometrics and the UK Research Assessment Exercise. In Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds.

Harnad, S. (2008) Self-Archiving, Metrics and Mandates. Science Editor 31(2) 57-59

Harnad, S. (2008) Validating Research Performance Metrics Against Peer Rankings. Ethics in Science and Environmental Politics 8 (11) doi:10.3354/esep00088 The Use And Misuse Of Bibliometric Indices In Evaluating Scholarly Performance

Harnad, S. (2009) Multiple metrics required to measure research performance. Nature (Correspondence) 457 (785) (12 February 2009)

Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives: Improving the UK Research Assessment Exercise whilst making it cheaper and easier. Ariadne 35.

Harnad, S., Carr, L. and Gingras, Y. (2008) Maximizing Research Progress Through Open Access Mandates and Metrics. Liinc em Revista.

---

[With many thanks to Professor Hal Abelson for the information]

---

OA Self-Archiving Policy: Massachussetts Institute of Technology (MIT) 

Full list of institutions

Massachussetts Institute of Technology (MIT) (US* institutional-mandate)
http://web.mit.edu/

Institution's/Department's OA Eprint Archives
[growth data] http://dspace.mit.edu/

Institution's/Department's OA Self-Archiving Policy

MIT Faculty Open-Access Policy

Passed by Unanimous of the Faculty, March 18, 2009

The Faculty of the Massachusetts Institute of Technology is committed to disseminating the fruits of its research and scholarship as widely as possible. In keeping with that commitment, the Faculty adopts the following policy: Each Faculty member grants to the Massachusetts Institute of Technology nonexclusive permission to make available his or her scholarly articles and to exercise the copyright in those articles for the purpose of open dissemination. In legal terms, each Faculty member grants to MIT a nonexclusive, irrevocable, paid-up, worldwide license to exercise any and all rights under copyright relating to each of his or her scholarly articles, in any medium, provided that the articles are not sold for a profit, and to authorize others to do the same. The policy will apply to all scholarly articles written while the person is a member of the Faculty except for any articles completed before the adoption of this policy and any articles for which the Faculty member entered into an incompatible licensing or assignment agreement before the adoption of this policy. The Provost or Provost's designate will waive application of the policy for a particular article upon written notification by the author, who informs MIT of the reason.

To assist the Institute in distributing the scholarly articles, as of the date of publication, each Faculty member will make available an electronic copy of his or her final version of the article at no charge to a designated representative of the Provost's Office in appropriate formats (such as PDF) specified by the Provost's Office.


The Provost's Office will make the scholarly article available to the public in an open- access repository. The Office of the Provost, in consultation with the Faculty Committee on the Library System will be responsible for interpreting this policy, resolving disputes concerning its interpretation and application, and recommending changes to the Faculty.

The policy is to take effect immediately; it will be reviewed after five years by the Faculty Policy Committee, with a report presented to the Faculty.

The Faculty calls upon the Faculty Committee on the Library System to develop and monitor a plan for a service or mechanism that would render compliance with the policy as convenient for the faculty as possible.