The Open Citation Project - Reference Linking for Open Archives

Three year evaluation plan

1  Introduction |  2  What we’re evaluating |  3  The evaluation plan
3.1 Focus |  3.2 Methodology |  3.3 Requirements |  3.4  Summary timetable |  3.5 Deliverables of the evaluation
 

1  Introduction

Project schedule
Year 1: October 1999 - September 2000
Year 2: October 2000 - September 2001
Year 3: October 2001 - September 2002

2  What we’re evaluating

Linking a reference to the document it cites is generally referred to as ‘backward linking’, as it points backward in time to an earlier document. Linking forward in time, to discover later documents to have cited a given paper, is also desirable (although in the faster-moving environment of eprint archives the distinction between forward and backward, before and after, can become blurred). Forward linking demands a more powerful database of papers and their references than has been built to date, but we plan a new version (v2.0) of the demonstrator with forward links and a better user interface in year 2.

References can also link to journal articles and other digital library services, not just to eprints. Developing link services of this type will depend on relationships with external partners and will not be implemented by the project alone, which is focussing on free-to-access repositories of academic papers. We have a number of such partners on our steering committee and hope to test links to their services in v2.0.

Additionally, at this stage we want to test an author deposit interface for archives, including a reference checking feature. It’s not clear what the target archive will be but the primary interface will be based on that used by the EPrints software. It must be a large archive, ideally one that has already adopted EPrints.

It is planned to extend these linking services to include other Open Archives in other disciplines (v2.5), notably those that support the OAi interoperability requirements. This is important because distributed archives need to enable services, such as linking services, to collect relevant data. The target area for this demonstration is likely to be computer science archives, since these are large and distributed (though some are disorganised). The largest computer science archives –the Networked Computer Science Technical Library (NCSTRL), and the Computing Research Repository (CoRR), will become OAi compatible, and there are various supplementary resources (including journal and conference publications) that the project has access to.

The blurring of forward and backward links leads us to consider the archives not in a time dimension but as a knowledge space. Addition of ‘knowledge-based’ linking services- e.g. using ontological reasoning to represent a sophisticated conceptual model of document terms and their relationships - will be the principal feature of the final project demonstrator (v3.0), which will attempt to inter-link the earlier version releases (v2.0 and v2.5), and other Open Archives where possible.

3  The evaluation plan

The evaluation plan will have formative, i.e. feeding back into the development of the project, and summative, i.e. outputs of the evaluation, elements.

3.1 Focus

• presentation of links (principally links on references; also knowledge-based links)
• effective integration of services (including online journals, other digital library services, etc.)
• the user interface (reference checking services)
• effective integration of resources (Open Archives, different disciplines)

3.2 Methodology

• specialist test user groups for each release selected from user and author lists for the linked archives – emphasis on presentation of links and integration of resources
• project steering group, includes archive administrators, developers, database providers, librarians, publishers, representatives of standards organisations and  learned societies – emphasis on integration of services
• special-interest groups, e.g. members of the Open Archives initiative, developers of reference linking services – especially for later releases
• local (Southampton) user group for pre-release usability testing of interfaces

3.3 Requirements

• Formal questionnaires (specialist and general users)
• Group meetings (steering group – may be geographically restricted)
• Individual meetings (local users)

3.4  Summary timetable

3.5  Deliverables of the evaluation

• Reports on each release to be produced for the steering group
• Dissemination within JISC
• Papers and presentations