Open Access Archivangelism

The UK Research Assessment Exercise has taken a few steps forward and a few steps back:

(1) In evaluating and rewarding the research performance of universities department by department, future RAEs (after 2008) will no longer, as before, assess only 4 selected papers per researcher, among those researchers selected for inclusion: All papers, by all departmental researchers, will be assessed. (Step forward)

(2) The assessment will be in terms of objective metrics, not just in terms of panel review. (Step forward)

(3) The metrics will be multiple, rather than just a single metric. (Step forward)

(4) The new system will apply to science, technology, and engineering, at least. (Step forward)

(5)The new system may only apply to science, technology and engineering. (Step Backward)

(6) The metrics considered may be only three, picked a priori: (i) prior research income, (ii) postgraduate numbers, and (iii) the "impact factor" (i.e., the average number of citations) of the journal in which each article was published. (Step Backward)

As I have pointed out many times before, (i) prior research income, if given too much weight, becomes a self-fulfilling prophecy, and reduces the RAE to a multiplication factor on competitive research funding. The result would be that instead of the current two autonomous components in the UK's Dual Support System (RAE and RCUK), there would only be one: RCUK (and other) competitive proposal funding, multiplied by the RAE metric rank, dominated by prior funding.

To counterbalance against this, a rich spectrum of potential metrics needs to be tested in the 2008 RAE, and validated against the panel review rankings, which will still be collected in the 2008 parallel RAE. Besides (i) research income, (ii) postgraduate student counts, and (iii) journal impact factors, there is a vast spectrum of other candidate metrics, including (iv) citation metrics for each article itself (rather than just its journal's average), (iv) download metrics, (v) citation and download growth curve metrics, (vi) co-citation metrics, (vii) hub/authority metrics, (viii) endogamy/interdisciplinarity metrics (ix) book citation metrics, (x) web link metrics, (xi) comment tag metrics, (xii) course-pack metrics, and many more.

All these candidate metrics should be tested and validated against the panel rankings in RAE 2008, in a multiple regression equation. The selection and weighting of each metric should be adjusted, discipline by discipline, rationally and empirically, rather than a priori, as is being proposed now.

Harnad, S. (2007) Open Access Scientometrics and the UK Research Assessment Exercise. In Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds.

(I might add that RCUK's plans to include "potential economic benefits to the UK" among the criteria for competitive research funding could do with a little more rational and empirical support too, rather than being adopted a priori.)

Stevan Harnad
American Scientist Open Access Forum