Explore Open Archives-Metalist | Evaluation | Tools and resources The Open Citation Project - Reference Linking and Citation Analysis for Open Archives

 

Evaluation

Version	Demonstrator	First available	Evaluation results
V. 2.1	Citation Analysis and Reference Linking demonstrator: Citebase	March 2002	December 2002
V. 1.0	Reference Linking demonstrator	May 2000	August 2000

V. 2.1 Citation Analysis and Reference Linking demonstrator: Citebase (March 2002)

• Find Cited and Linked full-text arXiv Documents (using Citebase, a citation-ranked search engine).
• e.g. most-cited papers with Stephen Hawking as an author

(Almost) the whole arXiv - linked! With reference links and citation analysis. The Opcit project links each of the 150,000+ papers in the arXiv physics archives to every other paper in the archives that it cites. Also - this is the clever part - discover what other papers cite a chosen paper.

Due to changes in the management of the underlying databases, V. 2.1 is a more robust and reliable version of the V. 2.0 Citation and Reference Linking demo, originally announced in August 2001.

• Evaluation forms (announced June 2002)
  • Form 1: Using Citebase, including sections about the evaluators, a practical exercise using Citebase, and user views of Citebase
  • Form 2: User Satisfaction Questionnaire, based on the well-known Software Usability Measurement Inventory (SUMI) form of questionnaire for measuring software quality from the end user's point of view

• Major results (first reported December 2002)
  • As exemplified by Citebase, Web-based citation indexing of open access eprint archives is closer to a state of readiness for serious use than had previously been realised
  • Within the scope of its primary components, the search interface and services available from its rich bibliographic records, Citebase can be used simply and reliably for the purpose intended
  • Citebase ompares favourably with other bibliographic services
  • Tasks can be accomplished efficiently with Citebase regardless of the background of the user
  • Better explanations and guidance are required for first-time users
  • Coverage is seen as a limiting factor, even though Citebase indexes over 200,000 papers from arXiv: non-physicists were frustrated at the lack of papers from other sciences.
  • Full evaluation report on Citebase

V. 1.0 Reference Linking demonstrator (May 2000)

• Find Linked arXiv Documents (using a simple search interface)
• Well-Linked Papers for non-physicists - examples of papers with lots of linked references

Papers from arXiv are converted to PDF for this demonstrator? Find out how and why.

• Results of V. 1.0 evaluation

Background to OpCit demonstrators

Evaluation of demonstrators produced by the project was based on a three-year evaluation plan produced in August 2000. Not all the demonstrators envisaged within the original plan were produced, due to problems accessing content from other data providers at the time. Instead the project concentrated on extending the citation indexing services provided by Citebase.

An earlier demonstration of reference linking, evaluated by the Open Journal Project and reported in this paper, also involved a number of partners from OpCit.

We are grateful to our colleagues at arXiv.org, and the maintainers of the arXiv mirror at Southampton University, for permission to use their services in this project.