The Open Citation Project - Reference Linking for Open Archives

Three year evaluation plan


Introduction |  2  What we’re evaluating |  3  The evaluation plan
3.1 Focus |  3.2 Methodology |  3.3 Requirements |  3.4  Summary timetable |  3.5 Deliverables of the evaluation

1  Introduction

Through collaboration between Southampton and Cornell Universities, and by building on the outputs of earlier projects, the Open Citation project is able to combine research into generic linking structures and interoperability requirements with early-stage implementations and evaluation from its first year. These preliminary, qualitative evaluations will be limited in scope and distribution, the purpose being to inform more complete and more sophisticated versions to follow in years 2 and 3. Usability tests and fuller evaluations will be performed on those versions.

Project schedule
Year 1: October 1999 - September 2000
Year 2: October 2000 - September 2001
Year 3: October 2001 - September 2002

2  What we’re evaluating

The project is developing and demonstrating reference linking services for eprint archives, notably those archives that conform to the Open Archives initiative (OAi). The principal resource is the Los Alamos physics archive, the largest and pre-eminent archive of its type, with over 100,000 documents. Initially a locally-stored version of that archive is being maintained as though it were a mirror but with all documents stored in a particular delivery format (pdf). This first demonstrator (v1.0) links every reference that points to another document that can be found in the archive to that document. This version is built and is currently being investigated.

Linking a reference to the document it cites is generally referred to as ‘backward linking’, as it points backward in time to an earlier document. Linking forward in time, to discover later documents to have cited a given paper, is also desirable (although in the faster-moving environment of eprint archives the distinction between forward and backward, before and after, can become blurred). Forward linking demands a more powerful database of papers and their references than has been built to date, but we plan a new version (v2.0) of the demonstrator with forward links and a better user interface in year 2.

References can also link to journal articles and other digital library services, not just to eprints. Developing link services of this type will depend on relationships with external partners and will not be implemented by the project alone, which is focussing on free-to-access repositories of academic papers. We have a number of such partners on our steering committee and hope to test links to their services in v2.0.

Additionally, at this stage we want to test an author deposit interface for archives, including a reference checking feature. It’s not clear what the target archive will be but the primary interface will be based on that used by the EPrints software. It must be a large archive, ideally one that has already adopted EPrints.

It is planned to extend these linking services to include other Open Archives in other disciplines (v2.5), notably those that support the OAi interoperability requirements. This is important because distributed archives need to enable services, such as linking services, to collect relevant data. The target area for this demonstration is likely to be computer science archives, since these are large and distributed (though some are disorganised). The largest computer science archives –the Networked Computer Science Technical Library (NCSTRL), and the Computing Research Repository (CoRR), will become OAi compatible, and there are various supplementary resources (including journal and conference publications) that the project has access to.

The blurring of forward and backward links leads us to consider the archives not in a time dimension but as a knowledge space. Addition of ‘knowledge-based’ linking services- e.g. using ontological reasoning to represent a sophisticated conceptual model of document terms and their relationships - will be the principal feature of the final project demonstrator (v3.0), which will attempt to inter-link the earlier version releases (v2.0 and v2.5), and other Open Archives where possible.

3  The evaluation plan

The plan is to evaluate the demonstrators produced by the project. There is currently no plan to evaluate the operation and implementation of the project itself.

The evaluation plan will have formative, i.e. feeding back into the development of the project, and summative, i.e. outputs of the evaluation, elements.

3.1 Focus

The nature of the project defines the following areas for evaluation:

3.2 Methodology

Feedback will be generated from these contributing groups: These are geographically broad groups, so questionnaires and electronic communication will of necessity be the main means of inviting feedback.

3.3 Requirements

The evaluation will require the following activities:

3.4  Summary timetable

Version Archives Added features Release period Evaluation
V1.0 arXiv physics archives Backward (in time) linking 2Q1 4Q1
V2.0 arXiv physics archives Forward linking, links to online journals services and other digital library services 2Q2 3Q2
V2.5 NCSTRL, CoRR, WWW conference series (ACM DL) Inter-link distributed archives 4Q2 1Q3
V3.0 All arXiv (inc. maths) and v2.5 (other Open Archives as appropriate) Knowledge linking 3Q3 4Q3

3.5  Deliverables of the evaluation

August 2000
<Home  Evaluation                                                                                                           
The OpCit Project
This page produced and maintained by the Open Citation project. Contact us