> Date: Tue, 19 Sep 2006 08:36:43 +0100
> From: "C.Oppenheim" <C.Oppenheim_at_LBORO.AC.UK>
> Subject: Re: Future UK RAEs to be Metrics-Based
>
> The correlation is between number of citations in total (and average number
> of citations per member of staff) received by a Department over the RAE
> period (1996-2001) and the RAE score received by the Department following
> expert peer review. Correlation analyses are done using Pearson or Spearman
> correlation coefficients. The fact that so few humanities scholars publish
> journal articles does not affect this result.
>
> A paper on the topic is in preparation at the moment.
>
> What intrigues me is why there is so much scepticism about the notion. RAE
> is done by peer review experts. Citations are also done by (presumably)
> experts who choose to cite a particular work. So one would expect a
> correlation between the two, wouldn't one? What it tells us is that high
> quality research leads to both high RAE scores AND high citation counts.
>
> I do these calculations (and I've covered many subject areas over the
> years, but not biblical studies - something for the future!) in a totally
> open-minded manner. If I get a non-significant or zero correlation in such
> a study in the future, I will faithfully report it. But so far, that hasn't
> happened.
While I do not challenge Charles' expertise or results in this matter - I
haven't looked at the specific recent articles, but there was some
well-supported work at the time of the previous RAE in 2001 - I do have a
serious question to ask about this.
What will be the impact amongst UK academics of making their funding (partly)
dependent on their citation rates?
It is the "law of unintended consequence" or, alternately, the lack of
consideration about "game playing" that makes the current RAE such a bad
method of deciding funding allocation. if you tell academics the rules of a
game, then they will generally be quite good at playing that game to win.
This skews the activity of academics away from their proper business unless
you can show that the way to win the game is to do the "correct" things
anyway.
Now, citation counts are derived from worldwide academia, but remember that
various other countries are also considering RAE-style measurements and some
are looking at metrics, while others seem to be mostly following the UK
example so we can assume they may well move to a similar metric-based system
sometime in the future.
It's all very well to claim that without citation-metric-based funding
decisions, that citation metrics mirror existing measurements. But what
evidence is there that this is a robust relationship under the assumption
that we move to a citation-metric measrement and away from the current system?
Note: this discussion may well be veering too far off-topic from OA policy
issues.
--
*E-mail*a.a.adams_at_rdg.ac.uk******** Dr Andrew A Adams
**snail*27 Westerham Walk********** School of Systems Engineering
***mail*Reading RG2 0BA, UK******** The University of Reading
****Tel*+44-118-378-6997*********** Reading, United Kingdom
Received on Wed Sep 20 2006 - 12:27:18 BST