The Open Citation Project - Reference Linking for Open Archives

Evaluating Citebase, an open access Web-based citation-ranked search and impact discovery service

IAM Group, Department of Electronics and Computer Science, University of Southampton, SO17 1BJ, United Kingdom
Contact for correspondence: Steve Hitchcock sh94r@ecs.soton.ac.uk

Version history of this report
This version 1.0, official report to JISC, released to selected users and evaluators December 2002
Version 2.0, edited for publication as a Technical Report, July 2003
Version 3.0, focus on usability results, edited for journal publication, draft, July 2003

This report was produced by the Open Citation Project, which was funded between 1999 and 2002 by the Joint NSF - JISC International Digital Libraries Research Programme.
 

Abstract

Summary

Other deliverables of the project, such as an application programming interface for reference linking, have been evaluated separately (Bergmark and Lagoze 2001). Another deliverable, EPrints.org software for creating open access Web-based archives (Gutteridge 2002), receives feedback from its already extensive list of registered implementers, which informs continuing development of new versions of the software (version 2.2 was announced in October 2002).

The OpCit evaluation of Citebase took the form of a two-part Web-based questionnaire, designed to test Citebase in two ways:

• Is it usable?
• Is it useful?

The evaluation was performed over four months from June to October 2002 by members of the Open Citation Project team at Southampton University. Observed tests of local users were followed by scheduled announcements to selected discussion lists for JISC and NSF DLI developers, OAI developers, open access advocates and international librarian groups. Finally, following consultation with our project partners at arXiv Cornell, arXiv users were directed to the evaluation by means of links placed in abstract pages for all but the latest papers deposited in arXiv.

A by-product of the exercise was raising awareness of Citebase among target user and prospective user groups, especially among arXiv physicists to whom the service had not previously been announced. By monitoring Web usage logs for Citebase it is shown that usage increased from around 25-45 visitors per day before the evaluation began to 660 daily visitors at its peak during the evaluation. Further, staging announcements to lists enabled response levels to be tracked for different user groups.

The principal finding is that Citebase fulfills the objective of providing a usable and useful citation-ranked search service. It is shown tasks can be accomplished efficiently with Citebase regardless of the background of the user. More data need to be collected and the process refined before it is as reliable for measuring impact.

The principle of citation searching of open access archives has thus been demonstrated and need not be restricted to current users..

Its deceptively simple search interface, however, masks a complexity that is characteristic of such services and which requires better explanations and guidance for first-time users.

Coverage is seen as another limiting factor. Although Citebase indexes over 200,000 papers from arXiv, non-physicists were frustrated at the lack of papers from other sciences. This is a misunderstanding of the nature of open access services, which depend on prior self-archiving by authors. In other words, rather than Citebase it is users, many of whom will also be authors, who have it within their power to increase the scope of Citebase by making their papers available freely from OAI-compliant open access archives. Citebase will index more papers and more subjects as more archives are launched. The evaluation thus highlights findings that need to be addressed in other forums, such as OAI and the Budapest Open Access Initiative. One of the announced objectives of the project was that Citebase, in highlighting both the unrealized potential of current levels of coverage, will stimulate further self-archiving by authors.

1 Introduction

It has been noted that while Garfield’s basic intentions were "essentially bibliographic", he has conceded that "no one could have anticipated all the uses that have emerged from the development of the SCI" (Guedon 2001). One of these uses is co-citation analysis (Small 1973), which makes possible the identification of emerging trends, or 'research fronts', which today can be visualised using powerful computational techniques (Chen and Carr 1999).

Another use, however, was to divert the SCI into a new business as a career management tool. As a result, Guedon claims that in "introducing elitist components into the scientific quest for excellence, SCI partially subverted the meaning of the science game".

New Web-based citation indexing services, such as ResearchIndex (also known as CiteSeer; Lawrence et al. 1999) and Citebase from the Open Citation (OpCit) Project, are founded on the same basic principles elaborated by Garfield (1994). Unlike Web of Knowledge which indexes core journal titles, these new services index full-text papers that can be accessed freely by users on the Web, and the indexing services are also currently free. While it is possible that open access indexing services founded on open access texts could re-democratise the role of citation indexing, there is no doubt these services will offer qualitatively different services from those provided by ISI: "Newer and richer measures of "impact" ... will be not only accessible but assessable continuously online by anyone who is interested, any time" (Harnad 2001). According to Lawrence (2001), open access increases impact.

According to Suber (2002) the "greatest benefit" of open access content services that are free to users will be "to provide free online data to increasingly sophisticated software which will help scholars find what is relevant to their research, what is worthy, and what is new". Citebase is an example of exactly that.

Despite the apparent advantage of open access, critical questions still have to be asked of these new services: are they useful and usable for the purposes of resource discovery and measuring impact? This report seeks to answer these questions based on an evaluation of Citebase, a citation-ranked search service. In the course of the investigation, some pointers to the resolution of these wider issues are also revealed.

2 Background to the evaluation

2.1 About Citebase

Just prior to the evaluation Citebase had records for 230,000 papers, indexing 5.6 million references. By discipline, approximately 200,000 of these papers are classified within arXiv physics archives. Thus, overwhelmingly, the current target user group for Citebase is physicists. The impact being made by OAI (Van de Sompel and Lagoze 2002) should help extend coverage significantly to other disciplines (Young 2002), through the emphasis of OAI on promoting institutional archives (Crow 2002). It is clear that a strong motivation for authors to deposit papers in institutional archives is the likelihood of subsequent inclusion in powerful resource discovery services which also have the ability to measure impact. For this reason there is a need to target this evaluation at prospective users, not just current users, so that Citebase can be designed for an expanding user base. Citebase needs to be tested and optimised for users, and nurtured by the communities that stand to benefit in the longer term.

The primary Citebase Web user interface (Figure 2.1) shows how the user can classify the search query terms (typical of an advanced search interface) based on metadata in the harvested record (title, author, publication, date). In separate interfaces, users can search by archive identifier or by citation. What differentiates Citebase is that it also allows users to select the criterion for ranking results by Citebase processed data (citation impact, author impact) or based on terms in the records identified by the search, e.g. date (see drop-down list in Figure 2.1). It is also possible to rank results by the number of 'hits', a measure of the number of downloads and therefore a rough measure of the usage of a paper. This is an experimental feature to analyse both the quantitative and the temporal relationship between hit (i.e. usage) and citation data, as measures as well as predictors of impact. Hits are currently based on limited data from download frequencies at the UK arXiv mirror at Southampton only.

The results shown in Figure 2.1 are ranked by citation impact: Maldacena's paper, the most-cited paper on string theory in arXiv at the time (September 2002), has been cited by 1576 other papers in arXiv. (This is the method and result for Q2.3 in the evaluation exercise described below.)

The combination of data from an OAI record for a selected paper with the references from and citations to that paper is also the basis of the Citebase record for the paper. A record can be opened from a results list by clicking on the title of the paper or on 'Abstract' (see Figure 2.1). The record will contain bibliographic metadata and an abstract for the paper, from the OAI record. This is supplemented with four characteristic services from Citebase:

• Graph of this Article's Citation/Hit History for the paper
• All Articles Cited by this Article (Reference List)
• Top 5 Articles Citing this Article (option to view All Articles Citing this Article)
• Top 5 Articles Co-cited with this Article (option to view All Articles Co-Cited with this Article)

Another option presented to users from a results list is to open a PDF version of the paper (see Figure 2.1). This option is also available from the record page for the paper. This version of the paper is enhanced with linked references to other papers identified to be within arXiv, and is produced by OpCit. Since the project began, arXiv has been producing referenced linked versions of papers. Although the methods used for linking are similar, they are not identical and OpCit versions may differ from versions of the paper available from arXiv. An important finding of the evaluation will be whether reference linking of full-text papers should be continued outside arXiv. An earlier, smaller-scale evaluation, based on a previous OpCit interface (Hitchcock et al. 2000), found that arXiv papers are the most appropriate place for reference links because users overwhelmingly use arXiv for accessing full texts of papers, and references contained within papers are used to discover new works. It was also reported that all aspects of the interface evaluated at that time needed more user-focussed development (see http://opcit.eprints.org/evaluation/v10/v10evaluation.html).

2.2 Scope of the evaluation

• The primary search interface
• Services available from a Citebase record
• Linked PDFs
• The effectiveness of the means of navigating between these services
• Support for users when these services fail to produce the required result

2.3 Purpose of the evaluation

1. evaluate the usability of Citebase (can it be used simply and reliably for the purpose intended)
2. assess the usefulness of Citebase (how does it compare and fit in with other services)
3. measure user satisfaction with Citebase
4. raise awareness of Citebase
5. inform ongoing development of Citebase

3 Description of the evaluation

3.1 Participants in the evaluation

3.2 Methods

• A two-part Web-based questionnaire
• Form 1: about users; a practical exercise - building a short bibliography; views of Citebase (http://www.ecs.soton.ac.uk/~aw01r/citebase/evalForm1.htm)
• Form 2: user satisfaction (http://www.ecs.soton.ac.uk/~aw01r/citebase/evalForm2.htm)
• Usage statistics for the Citebase search page (http://citebase.eprints.org/usage/)

After removing blanks, duplicates and test submissions, a total of 195 valid submissions of Form 1 were received. Of these users, 133 also completed Form 2, which was linked from the submit button of Form 1.

3.3 Timescale

There is scope for a follow-up exercise, using the email addresses supplied by over 100 interested users, to test the effectiveness of changes to Citebase following the evaluation.

3.4 Discussion of the methods

As already indicated, Citebase is aimed at a much wider user group, both now and in the future, and the evaluation had to be extended to a representative section of those users. Open invitation is one way of achieving this. There are drawbacks to inviting evaluation based on a Web-only questionnaire, most obviously the lack of direct contact with users, and the consequent loss of motivation and information. Balancing this should be simplicity, easy accessibility and continuous availability. Web surveys have widened use and reduced the costs of survey techniques, but introduce new complexities (Gunn 2002). Efforts were made to ensure the forms were usable, based on the observed tests, and that Citebase offered a reliable service during the period of the evaluation. Availability of forms and service were monitored and maintained as far as was possible during the period of evaluation (see section 7.3).

A perennial problem with forms-based evaluation, whether users are remote or not, is that badly designed  forms can become the object of the evaluation. In tests of this type, where most users are experiencing a service for the first time, observation suggests that users may have understood the service more intuitively had they just looked at it as a search service rather than being introduced to it via step-by-step questions. This raises the question of whether the service to be evaluated, Citebase, should have been promoted more extensively prior to the evaluation. This would have increased familiarity, but it was felt this would make it more difficult to attract users to the evaluation unless those users were being brought to Citebase via the evaluation.

In contrast to Web forms, usage logs are an impeccable record of what people actually do, although there are problems of interpretation, and there are no standards for the assessment of Web logs.

The response to the evaluation from arXiv physicists, the primary target user group, was a little below expectations, although replies from other users were higher than expected. Our earlier survey received nearly 400 replies from arXiv users (Hunt 2001). Similarly, usage of Citebase among the same group might have been expected to be higher.

It is likely the lower number of respondents to the evaluation was due to the method of linking from arXiv to the evaluation. For the earlier survey, arXiv linked directly from a notice on its home page to the Web form. In this case abstract pages for papers in arXiv linked to the corresponding Citebase records. To get users to the evaluation form required that a linked notice be inserted temporarily in the Citebase records (Figure 3.1).

As a means of bringing arXiv users to Citebase on an ongoing basis, this is an ideal, task-coupled arrangement. From the perspective of the evaluation, however, users were expected to follow two links to reach the evaluation, and were thus required to take two steps away from their original task. Since there was no direct link to the evaluation from the arXiv home page, and therefore no prior advocacy for, or expectation of, Citebase or the evaluation. perhaps it should not be surprising that the response did not match our earlier survey.

Usage of Citebase would have been affected for the same reason; also by a prominent notice:

placed alongside the new links to Citebase in arXiv (Figure 1 in Hitchcock et al. 2002).

It is planned that the Citebase developer (Tim Brody) will work with our arXiv partners to refine Citebase in response to the evaluation with a view to removing the warning notices and increasing advocacy for the service. It will be instructive to see how usage levels change as a result of such changes.

4 Preparing Citebase for evaluation

This was resolved prior to open Web evaluation. According to Tim Brody, Citebase developer: "A few database tweaks later (!) and it's now ~0.5s to return the co-cited papers. It is slightly different behaviour now: whereas before it ranked (in descending order) co-citedness, paper impact, author impact, it now only ranks by co-citedness (avoids a table join)."

The other area of concern for Citebase were the descriptions, support and help pages, a vital part of any new and complex service. There was some reorganisation of this material and new pages added. This is an ongoing process and will continue to be informed by users.

Terminology was another aspect raised leading up to the evaluation. Terms used in the evaluation form such as "most cited" can be interpreted as the largest number of citations for an author or the largest average number of citations per paper for the same author. On the form this was revised (Q2.1). More generally, efforts were made to make terminology in Citebase comparable with ISI.

If bibliographic tools have been subverted, whether by design or not, to serve as career management tools, there is no hiding from the fact that new, experimental services will produce contentious results. This was a particularly acute concern during the preparation of Citebase for testing. A warning notice was added prominently to the main search page:
 

Citebase was incomplete during the evaluation because new arXiv papers and their references were not harvested once the evaluation began in June. It was decided the data should be static during the evaluation, to ensure all users were evaluating the same object (some minor changes were made during the evaluation period, and these are highlighted in section 4.1). In arXiv, papers with numbers before 0206001 (June) had a link to Citebase, but not those deposited after.

Also, not all references could be extracted from all papers, which clearly would affect impact results. Techniques and software for automated reference extraction have been discussed by Bergmark (2000). Since the evaluation closed Citebase data have been brought up-to-date, and the reference parsing algorithm has been refined to improve extraction rates.

Warnings were also strengthened, after much discussion, around the hit data graphs displayed in Citebase records (Figure 4.1). Reservations about this feature were expressed by arXiv Cornell colleagues, for the following reasons:

• It is a departure from a long-standing arXiv policy not to release access data
• We are aware the doing this analysis correctly is not at all easy
• Even if the analysis is done correctly, the results often have little meaning without some greater understanding.
• The data are for one mirror
• People would attempt to skew the data artificially to get a better 'rating'

4.1 Updating Citebase during the evaluation

5 Design of the evaluation forms

• determine the context of the user
• present a practical exercise
• invite the user's views on specific features and more general views
• sign-up users for a follow-up exercise

5.1 User context

Similarly, since Citebase will extend coverage to new OAI archives, it is helpful to know the level of awareness of OAI among evaluators (Q1.4) , and whether they use other OAI services (Q1.5).

As with all other sections on the evaluation forms, this section ends by inviting open comments from evaluators, which can be used to comment on any aspect of the evaluation up to this point.

5.2 Practical exercise

At this point users were prompted to open a new Web browser window to view the main Citebase search interface. It was suggested this link could have been placed earlier and more prominently, but this was resisted as it would have distracted from the first section.

Questions 2.1-2.3 involved performing the same task and simply selecting a different ranking criterion in the search interface (Figure 2.1). Selectable ranking criteria is not a feature offered by popular Web search engines, even in advanced search pages, which the main Citebase search page otherwise resembles. The user's response to the first question is therefore important in determining the method to be used, and Q2.1 might be expected to score lowest as familiarity increases for Q2.2 and Q2.3. Where Q2.1 proved initially tricky, however, observed tests revealed that users would return to Q2.1 and correct their answer. We have no way of knowing to what extent this happened in unobserved submissions, but allowance should be made for this when interpreting the results.

The next critical point occurs in Q2.4, when users are effectively asked for the first time to look below the search input form to the results listing for the most-cited paper on string theory in arXiv (Q2.3). To find the most highly cited paper that cites this paper, notwithstanding the apparent tautology of the question, users have to recognise they have to open the Citebase record for the most-cited paper by clicking on its title or on the Abstract link. Within this record the user then has to identify the section 'Top 5 Articles Citing this Article'. To find the paper most often co-cited with the current paper (Q2.5) the user has to scroll down the page, or use the link, to find the section 'Top 5 Articles Co-cited with this Article'.

Now it gets slightly harder. The evaluator is asked to download a copy of the full-text of the current paper (Q2.6a). What the task seeks to determine is the user's preference for selecting either the arXiv version of the paper of the OpCit linked PDF version. Both are available from the Citebase record. A typical linked PDF was illustrated by Hitchcock et al. (2000). Originally the Citebase record offered a 'linked PDF', but during the evaluation the developer changed this to a PDF graphic (Table 4.1). The significance of omitting 'linked' is that this was the feature differentiating the OpCit version. Given that it is known physicists tend to download papers in Postscript format rather than PDF (http://opcit.eprints.org/ijh198/3.html), it is likely that a simple PDF link would have little to recommend it against the link to the arXiv version.

As a check on which version users had downloaded, they were asked to find a reference (Q2.6b) contained within the full text (and which at the time of the evaluation was not available in the Citebase record, although it appeared in the record subsequently). To complete the task users had to give the title of the referenced paper, but this is not as simple as it might be because the style of physics papers is not to give titles in references. To find the title, the user would need to access a record of the referenced paper. Had they downloaded the linked version or not? If so, the answer was one click away. If not, the task was more complicated. As final confirmation of which version users had chosen, and how they had responded subsequently, users were asked if they had resorted to search to find the title of the referenced paper. In fact, a search using Citebase or arXiv would not have yielded the title easily.

In this practical exercise users were asked to demonstrate completion of each task by identifying an item of information from the resulting page, variously the author, title or URL of a paper. Responses to these questions were automatically classified as true, false or no response. Users could cut-and-paste this information, but to ensure false responses were not triggered by mis-keying or entering an incomplete answer, a fuzzy text matching procedure was used in the forms processor.

Although this is an indirect measure of task completion, the results of this exercise can be read as an objective measure showing whether Citebase is a usable service. As an extra aid to judge the efficiency with which the tasks are performed, users were asked to time this section. One idea was to build a Javascript clock into the form, but this would have required additional user inputs and added to the complexity of the form.

5.3 Views on Citebase

Questions 3.1 and 3.2 enquire about Citebase as it is now and as it might be, respectively. It is reasonable to limit choices in the idealised scenario (Q3.2) so that users have to prioritise desired features. Users are likely to be more critical of the actual service, so it seems safe to allow a more open choice of preferred features.

Citebase has to be shaped to offer users a service they cannot get elsewhere, or a better service. Q3.3 seeks to assess the competition. This part of the evaluation is concluded by asking the user for a view on Citebase, not in isolation, but in comparison with familiar bibliographic services.

5.4 Follow-up and submission

There is a second part to the evaluation, which is displayed to users automatically on submission of Form 1. It became apparent from observed tests that users do not always wait for a response to the submission and may miss Form 2, so a clear warning was added above the submission button on Form 1.

On submission the results were stored in a MySQL database and passed to an Excel spreadsheet for analysis.

5.5 Response and Form 2

Form 1 prompted users to respond to specific questions and features, and gives an impression of their reaction to the evaluated service, but does not really explore their personal feelings about it. A recommended way of tackling this is an approach based on the well-known Software Usability Measurement Inventory (SUMI) form of questionnaire for measuring software quality from the end user's point of view. Form 2 is a short implementation of this approach which seeks to discover:

• Impression - the user's feelings when using Citebase system
• Command - the measure to which the user feels in control
• Effectiveness - the degree to which users feel they can complete the task while using Citebase
• Navigability - the degree to which the user can move around the application

Four response options, ranging from very positive to very negative, are offered for each of four statements in each section. These responses are scored -2 to 2. A neutral response is not offered, but no response scores zero. A statement that users typically puzzle over is 'If the system stopped working it was not easy to restart it', before choosing not to respond if the system did not fail at any stage. Users often query this, but an evaluation, especially where users are remote from testers, has to anticipate all possible outcomes rather than make assumptions about reliability.

6 Observed testing

Scenario: Users worked with machines in their own environment. Users were assured that they were not being tested, but that the system was being tested. Once they were in front of a machine with a working Web browser and connection, they were handed a printed copy of the evaluation forms as an aid and for notes, not instead of the online version. They were then given the URL to access evaluation Form 1, with no other instruction. Once started, observers were to avoid communication with users. Users were debriefed after completing the tasks.

Main findings (actions):

• Questions 2.1, 2.4 and 2.6 were confirmed to be the critical points in the practical exercise.
• There was a tendency to self-correction of the initial questions in section 2. The tests were performed carefully and watchfully, with steps being re-traced to correct some answers in section 2 as the method of using Citebase became clearer.
• Non-physicist users found section 2 variously "difficult" and "unfamiliar". Questions 2.4 and 2.6 were found to be tricky, and user-pairs spent a lot of time reading help and explanation sections, or simply discussing the right approach to use.
• At the time of the observed evaluations Citebase was very slow. It could take over a minute to download the page for Q2.4 (see section 4). Some users were tempted to start the next question before finishing the previous one, and this resulted in more confusion and more time spent on the exercise. (Action: the Citebase database was restructured, returning pages in this test in less than 0.5 s)
• Not all users had Acrobat readers installed on their machines and were not immediately able to view linked PDFs (Q2.6a-b). This is a usability issue for Citebase to address rather than this exercise.
• One group of users was unable to download the PDF version not because Acrobat was missing but because of the speed of the system.
• Users were unsure if the comment boxes in each section were optional (this was not amended as it was not our intention to dissuade users from commenting where necessary).
• For Q3.1 there appeared to be a tendency for users to tick boxes/options that were familiar from the previous exercise.
• After submitting Form 1 users were happy and relieved to see the congratulatory message with answers to section 2, but were unprepared for Form 2 that followed. (Action: the notice for Form 2 was capitalised in the first line of the concluding paragraph in Form 1, just above the submit button.)
• The text on the first half of the page responding to submission of Form 1 was too small. (Action: text enlarged.)
• There was relief at the conclusion. The team-based observed format of the tests were clearly quite demanding on users.

7 Usage: Citebase and the evaluation

7.1 Open announcements: effect on the evaluation

• colleagues in digital library research programmes (OpCit steering group, NSF-DLI2 lists), also
• JISC-Development http://www.jiscmail.ac.uk/cgi-bin/wa.exe?A2=ind0207&L=jisc-development&T=0&F=&S=&P=1332
• advocates of open access to the scholarly literature (BOAI, eprints-underground lists), also
• FOS-Forum http://www.topica.com/lists/fos-forum/read/message.html?mid=904790801
• OAI-general http://www.openarchives.org/pipermail/oai-general/2002-July/000187.html
• September98-Forum http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2121.html
• librarians (arl-ejournal (US librarians), Lib Serials lists), also
• notice in D-Lib Magazine July/August issue http://www.dlib.org/dlib/july02/07inbrief.html#HITCHCOCK
• lis-scitech (UK librarians) http://www.jiscmail.ac.uk/cgi-bin/wa.exe?A2=ind0207&L=lis-scitech&D=1&T=0&O=D&F=&S=&P=1067

Figure 7.1. Chart of daily responses to evaluation Form 1(July-November 2002)

Figure 7.2. Citebase usage: summary statistics for July 2002 showing number of distinct visits (yellow chart) to citebase.eprints.org. See http://citebase.eprints.org/usage/usage_200207.html for full chart and data (excludes all hits from soton.ac.uk and from cs.odu.edu (DP9), but includes search engines)

It can be seen that the highest response to the evaluation during the period of open announcements occurred between 12-14 July following announcements to open access advocates (Figure 7.1), but Citebase usage in July was highest on the 22nd (Figure 7.2) after announcements to library lists.
 

Table 7.1: Citebase usage spikes (unique site visits) attributed to list announcements

Date (July)	No. of visits	Suspected source of users
22nd	207	Delayed reaction to library mails over a weekend
12th	175	OAI, Sept-Forum, FOS-Forum
15th	159	D-Lib Magazine
29th	138	PhysNet?
8th	109	Possibly delayed reaction to JISC, DLI mails over a weekend

7.2 ArXiv links to Citebase: bringing physicists to the evaluation

Figure 7.3. Citebase usage: summary statistics for August 2002 showing number of distinct visits (yellow chart) to citebase.eprints.org. See http://citebase.eprints.org/usage/usage_200208.html for full chart and data (excludes all hits from soton.ac.uk and from cs.odu.edu (DP9), but includes search engines)

The impact of arXiv links on usage of Citebase was relatively much larger than that due to list announcements, as can be seen in Figure 7.4 in the column heights for July (list announcements) against August, September and October (arXiv links) (ignoring the red chart which emphasises the effect of Web crawlers rather than users).

Figure 7.4. Usage statistics for citebase.eprints.org. See http://citebase.eprints.org/usage/ for full chart and data (excludes all hits from soton.ac.uk and from cs.odu.edu (DP9), but includes search engines)
*image saved on 15 November

Table 7.2 puts the growth of Citebase usage (by visits) in perspective over this period, prior to the evaluation (February-June), due to list announcements (July), due to new arXiv links (August), and during the first full month of arXiv links (September).
 

Table 7.2. Growth of visits to Citebase, February-September 2002 (yellow columns in usage charts)

	February-June	July	August	September
Average daily visits	25-45	85	211	402
Highest daily visits	95 (8th May)	207 (22nd)	660 (22nd)	567 (4th)

The effect of the arXiv links on the evaluation were materially different, however, because the links were to Citebase, and only indirectly from there to the evaluation (see section 3.4). Table 7.3 shows how efficiently Citebase users were turned into evaluators on the best days for submission of evaluation Form 1. It shows how list announcements taking users directly to the evaluation returned the highest percentage of daily submissions from all Citebase users. Although overall usage of Citebase generated by arXiv links was much larger than for list announcements, this was not effectively translated into more submissions of the evaluation.
 

Table 7.3: Turning Citebase users into evaluators

Date	No. of evaluation forms returned (Figure 7.1)	Percentage of Citebase visitors that day
July 12th	16	9.1
July 13th	8	6.1
July 15th	7	4.4
July 22nd	7	3.4
July 8th	6	3.4
August 21st	6	1.3
August 23rd	6	1.1
August 27th	6	1.0
August 22nd	6	0.9

ArXiv.org HTTP server daily usage (http://arxiv.org/show_daily_graph) shows c.15,000 hosts connecting each day, i.e. approximately 3.3% of arXiv visitors become Citebase users. The challenge for Citebase, highlighted by these figures, is to attract a higher proportion of arXiv users by examining the results of this evaluation to improve the service Citebase offers. What isn't yet known is what proportion of arXiv usage is mechanical downloads, just keeping up with the literature, to be read later offline. Citebase will make little difference to this type of activity, but instead will help more active users, and here its proportionate share may already be much higher.

7.3 Network access and Citebase outages

• Wed, 7 Aug 2002 15:49:01 Citebase unavailable for 2-3 hours due to the database server being moved.
• Fri, 9 Aug 2002 15:22:04 Problems arising from above move. The database was restarted Sunday PM
• 16 Sep 2002 11:49:28 Virus attack on machine running Citebase. Machine taken off the network as precaution. Network access to Citebase restored 17 Sep 2002 11:08:05

8 Results: Form 1 - using Citebase

8.1 About the evaluators

Other users included: Librarians (13), Cognitive scientists (10), Biologists (5), Cognitive neuroscientists (3), Health scientists (3), Medical psychiatrists (2), Sociologists (2), Publishers (2), and a teacher, information professional, behavioural geneticist, media specialist, philosopher, geomorphologist, engineer, economist, technical marketer, undergraduate.

Q1.2 Have you used the arXiv eprint archive before?
 

Table 8.1. Usage of arXiv (physicists only)

Daily	Regularly	Occasionally (less than monthly)	No
56 (50)	26 (11)	28 (3)	79 (5)

Q1.3 If you have used arXiv, which way do you access arXiv papers? (you may select one or more)
 

Table 8.2. Means of accessing arXiv papers (physicists only)

Web search or other Web services	Don't use arXiv	Daily or regular browsing	Reference links in papers	Email alerts from arXiv	Bibliography or library services	Email alerts from other services	No response
37 (18)	29 (1)	65 (47)	31 (18)	25 (20)	16 (6)	5 (1)	58 (5)

Q1.4 Had you heard of the Open Archives Initiative?
 

Yes	No
99 (11)	86 (55)

( ) physicists only
 

a

b

Figure 8.3 Correlations with prior knowledge of OAI: a, with subject discipline (x axis: physics=4, maths=3, computer=2, infoScience=1, other=0), physicists are least likely, information scientists most likely to have heard of OAI, correlation= -0.46758, N=189, p<0.001; b, with level of arXiv usage (x axis: daily usage=4, regular usage=3, occasional usage=2, no usage=1), those who use arXiv least or not at all are more likely to have heard of OAI, correlation=-0.35072, N=189, p<0.001 (y axis: 2=heard of OAI, yes, 1=have not heard of OAI)

Q1.5 Have you used any other OAI services? (you may select one or more)
 

arc

8

myOAI

10

kepler

6

Other

2

No response

178

User comments on this section (highlights)

As usual, I find myself an "outsider" in discussions of things that will be important to me very soon. I find there is no category for me to go into. You guys need to look beyond geekdom to think about ordinary social scientists, librarians, educationists

In addition to CogPrints I am a heavy user of the CiteSeer ResearchIndex service. The automatic citation linkage adds a lot of value to that service.

8.2 Commentary on Citebase evaluators

Among non-scientists, as the comment above indicates, there may be a sense of exclusion. This is a misunderstanding of the nature of open access archives and services. No disciplines are excluded, but services such as Citebase can only act on major archives, which currently are mostly in the sciences. The primary exception is economics, which has distributed archives indexed by RePEc (http://repec.org/). There are plans to index RePEc in Citebase.

It is true, if this is what is meant by the "outsider" above, that Q1.1 in this evaluation anticipated that evaluators would mostly be scientists of certain types, as shown in Figure 8.1. It must also be added that the Citebase services of impact-based scientometric analysis, measurement and navigation are intended in the first instance for research-users, rather than lay-users, because the primary audience for the peer-reviewed research literature is the research community itself.

About a third of evaluators were physicists, although the number of physicists as a proportion of all users might have been expected to be higher given the concentration of Citebase on physics. (Physicists can be notoriously unfond of surveys, as the ArXiv administrators warned us in advance!)

Physicists in this sample tend to be daily users of arXiv (Q1.2). Non-physicists, noting that arXiv has smaller sections on mathematics and computer science, tend to be regular or occasional users of arXiv (Figure 8.1). Beyond these disciplines most are non-users of arXiv, and thus would be unlikely to use Citebase given its present coverage.

Most arXiv users in this study access new material by browsing, rather than by alerts from arXiv (Q1.3). The relatively low ranking of the latter was unexpected. There is some encouragement for services such as Citebase (note, at this stage of the evaluation users have not yet been introduced to Citebase) in the willingness to use Web search and reference links to access arXiv papers (second and third most popular categories of access). It is possible, as mentioned above, that the Citebase evaluators were a biased sample of arXiv daily users who do not download mechanically.

OAI is familiar to over half the evaluators, but not to many physicists (Q1.4, Figure 8.3a). The latter is not surprising. OAI was originally motivated by the desire to encourage researchers in other disciplines to build open access archives such as those already available to physicists through arXiv, although the structure of Open Archives, unlike arXiv, is de-centralised (Lynch 2001).

Although OAI has had an impact among most non-physicist evaluators - again probably preordained through list selection - there is clearly a problem attracting these users to OAI services (Q1.5). Either current OAI services are not being promoted effectively, or they are not providing services users want -- or this may be merely a reflection of the much lower availability of non-physics OAI content to date! As an OAI service, this result shows the importance for Citebase of learning the needs of its users from this evaluation, and of continuing to monitor the views of users. More generally, this result suggests there are stark issues for OAI and its service providers to tackle. To be fair, the services highlighted on the questionnaire are mainly research projects. It is time for OAI services to address users.

8.3 Practical exercise: building a short bibliography

( ) physicists only Q2.1 Who is the most-cited (on average) author on string theory  in arXiv? Correct 141 (45) Incorrect 20 (8) No answer 34 (15) Q2.2 Which paper on string theory is currently being browsed most often in arXiv? Correct 133 (41) Incorrect 16 (8) No answer 46 (19) Q2.3 Which is the most-cited paper on string theory in arXiv? Correct 145 (48) Incorrect 9 (2) No answer 41 (18) Q2.4 Which is the most highly cited paper that cites the most-cited paper above? (critical point) Correct 122 (44) Incorrect 26 (5) No answer 47 (19) Q2.5 Which paper is most often co-cited with the most-cited paper above? Correct 133 (46) Incorrect 12 (3) No answer 50 (19) Q2.6a  Download the full-text of the most-cited paper on string theory. What is the URL? Correct 124 (42) Incorrect 13 (3) No answer 58 (23) (Correct=Opcit linked copy 71 (15) +arXiv copy 53 (27)) Q2.6b In the downloaded paper, what is the title of the referenced paper co-authored with Strominger and Witten (ref [57])? Correct 105 (35) Incorrect 27 (9) No answer 63 (24) Q2.6c Did you use search to find the answer to 2.6b?   No 118 (40); Yes 18 (3)

Figure 8.4. All users: progress in building a short bibliography through Q2.1-2.6b in evaluation Form 1 (T=true, F=false, N=no response) Figure 8.5. Physicists only: progress in building a short bibliography through Q2.1-2.6b in evaluation Form 1 (T=true, F=false, N=no response)

Time taken to complete section 2
 

1-5 minutes	5-10	10-15	15-20	20-25	25-30	30+	?	Total
13 (6)	60 (21)	36 (14)	17 (5)	9 (0)	6 (0)	2 (0)	5 (2)	147

a

b

Figure 8.6 Correlations between time taken to complete section 2 and: a, subject disciplines (x axis: physics=4, maths=3, computer=2, infoScience=1, other=0) correlation= -0.15, N=140, p<0.077, b, level of arXiv usage (x axis: daily usage=4, regular usage=3, occasional usage=2, no usage=1) correlation=-0.18, N=140, p<0.033

User comments on this section (highlights)

I ran into two problems doing the task: (1) it wasn't clear at all initially how to find the most cited author (to do that task you'd have to go into the dropdown) -- I ended up doing the tasks backward and managed to find that feature in the dropdown when I was doing something else. (2) it was a little hard to find the paper given three authors -- I couldn't find the paper searching by author or by its hep-th number and ended up having to look it up in arxiv.org.

The site is very weak in explanations on what is possible. Perhaps experts would know what they were doing but for me this was very difficult.

Help documentation failed to help me in most cases.

Your questions too easily lead to the answers.

You need clear definitions of your terms in standard English. 'Author hits', and 'Paper Hits'

Fascinating ability to cross-check links between papers, though I would need more practice to not be confused as to where I was!

I think it would be more accurate if the search engine would accept "string theory". In the current case, it seems that some papers irrelevant to string theory are retrieved, because they are relevant either to string or to theory.

Easy to do & that usually means well thought out.

Extremely informative, a good way to get people to learn about the handy features of the system.

hmmm... there must be a difference between "rank by cocitedness" and the "cited with" button, but i can't guess what it is.

8.4 Commentary on building a short bibliography using Citebase

Reinforcing these results, almost 90% of users (100% of physicists) completed the exercise within 20 mins, with approximately 50% (55% physicists) finishing within 10 mins. There appears to be some correlation between subject disciplines and level of arXiv usage with the time taken to complete the exercise (Figure 8.6), although neither correlation is statistically significant. Taken together these results show that tasks can be accomplished efficiently with Citebase regardless of the background of the user.

As anticipated, Q2.4 proved to be a critical point, showing a drop in correct answers from Q2.3. The upturn for Q2.5 suggests that user confidence returns quickly when familiarity is established for a particular type of task. Similarly, the highest number of correct answers for Q2 .3 shows that usability improves quickly with familiarity of the features of a particular page. At no point in either Figure 8.4 or 8.5 is there evidence of a collapse of confidence or of willingness among users to complete the exercise.

On the basis of these results there can be confidence in the usability of most of the features of Citebase, but the user comments in this section draw attention to some serious usability issues - help and support documentation, terminology - that must not be overshadowed by the results.

The incidental issue of which PDF version users prefer to download, OpCit or arXiv version (Q2.6a), was not conclusively answered, and could not be due to the change in format on the Citebase records for papers (Table 4.1). It can be noted that among all users, physicists displayed a greater preference to download the arXiv version.

8.5 User views of Citebase

Q3.2 What would most improve Citebase?
 

Table 8.4: Improving Citebase (physicists only)

Wider coverage	Better explanations	Faster results	Better interface	More links	More papers	Other Responses	No response
50 (22)	30 (7)	3 (1)	11 (5)	6 (0)	30 (12)	24 (6)	41 (15)

Q3.3  What services would you use to compile a bibliography in your own work and field? (you may select one or more)
 

Table 8.9: Compiling a bibliography (physicists only)

My own bib list	Online library services	Web services	Offline library services	Other	No response
84 (31)	85 (23)	91 (37)	24 (5)	0	49 (14)

Q3.4 How does Citebase compare with these bibliography services (assuming that Citebase covered other subjects to the degree it now covers physics)?
 

Table 8.8: Comparing Citebase with other bibliography services

Very favourably	Favourably	No response	Unfavourably	Total
28 (7)	98 (40)	54 (16)	15 (5)	195

User comments on this section (highlights)

• If it was clear (to me) what the database was doing then it would be quite useful for searching, and finding seminal papers in a field.
• Very clever and valuable resource.
• With better explanations and an improved interface this would be an excellent service.
• Great to have this facility for eprints - might make people more happy to use them which would be a good thing!
• Looks great -- would love to see something similar in the human sciences
• The subject coverage is too narrow. Would need more engineering and chemistry coverage.
• The most important improvement is online access to the full text.
• The coverage is still too small to give me a reliable overview, especially in fields bordering and including engineering. But the features that are implemented have impressed me tremendously.
• I do not think this database is a good idea in general. It only lists papers in or related to ArXiv which is a very much incomplete source of information in mathematics. I think using it in any way to evaluate someone's work will be misleading
• Data base too small. Should contain all other OAI data providers containing physics, PhysDoc, etc.
• Very noisy results, missing articles. No match today of services like inspec or science citation index. The idea of course is very good, and I hope the quality improves.

8.6 Commentary on user views of Citebase

Users found it harder to say what would improve Citebase, judging from the number of 'no responses' to Q3.2. Wider coverage, especially in terms of more papers, is desired by all users, including physicists. The majority of the comments are complaints about coverage.

Signs of the need for better support documentation reemerge in this section. Although the number of users calling for a better interface is not high, comments indicate that those calling for improvements in this area are more vociferous. Among features not offered on the questionnaire but suggested by users, the need for greater search precision stands out.

There is a roughly equal likelihood that users who participated in this survey will use Web services (e.g. Web search), online library services and personal bibliography software to create bibliographies (Q3.3). This presents opportunities for Citebase to become established as a Web-based service that could be integrated with other services. The lack of a dominant bibliography service, including services from ISI, among this group of users emphasises the opportunity.

Citebase is beginning to exploit that opportunity (Q3.4), but needs to do more to convince users, even physicists, that it can become their primary bibliographic service.

Attempts to correlate how Citebase compared with other bibliographic services with other factors considered throughout the evaluation - with subject discipline, with level of arXiv usage, and time taken to complete section 2 - showed no correlations in any case (Figures 8.11-8.13). This means that reactions to Citebase are not polarised towards any particular user group or as a result of the immediate experience of using Citebase for the pre-set exercise, and suggests that the principle of citation searching of open access archives has been demonstrated and need not be restricted to current users.

There is little opportunity in this section for users for users to compare, contrast and discuss features of Citebase that differentiate it from other services. In particular, Citebase offers access to full texts in open access eprint archives. It is an aspect that needs to be emphasised as coverage and usage widen. Comments reveal that some users appreciate this, although calls for Citebase to expand coverage in areas not well covered now suggest this is not always understood. It is not possible for Citebase to simply expand coverage unless there is recognition and prior action by researchers, as authors, of the need to contribute to open access archives. One interpretation is that users in such areas do not see the distinction between open access archives and services and paid-for journals and services, because they do not directly pay for those services themselves - these services appear to be free.

8.7 Follow-up and comment on the evaluation

User comments on the evaluation: when the form becomes the object

• The questions are quite comprehensive & concise
• Well-thought out interactive test
• Too long an exercise and questionnaire
• This is really well done as an evaluation

Submission of Form 1

On submission of Form 1 users see a message thanking them for their participation and congratulating them on compiling a bibliography on string theory and giving the answers to the questions in section 2. Based on comments submitted, many users were keen to have their answers confirmed, although the response was not personalised in terms of telling users whether they were right or wrong, just what the correct answers were (see Appendix 1).

9 Results: Form 2 - user satisfaction with Citebase

Form 2 could have been longer and explored other areas, but this may have inhibited the number of responses. As Form 2 was separate from Form 1 it was not expected that all users would progress this far. Of 195 users who submitted the first form, 133 completed Form 2 (http://www.ecs.soton.ac.uk/~aw01r/citebase/evalForm2.htm). Details of the submissions to Form 2 can be viewed as an Excel spreadsheet saved as a Web page (again with the proviso that this is best viewed using an up-to-date or recent version of Internet Explorer).

The summary results by question and section are shown in Table 9.1 and Figure 9.1.
 

Table 9.1. Satisfaction scores (Form 2)

Question	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
Average score by Q.	0.92	0.79	1.39	1.05	0.41	1.17	0.83	1.02	0.65	1.07	1.42	0.99	0.92	0.27	0.57	0.26
Section	Impression				Command				Effectiveness				Navigability
Average by section	1.04				0.86				1.03				0.51

The highest score was recorded for Q11, indicating that on average users were able to find the information required most of the time. Scoring almost as high, Q3 shows users found the system frustrating to use only some of the time.

The questions ranked lowest by score, Q14 and Q16, suggest that users agree weakly with the proposition that there were plenty of ways to find the information needed, and disagreed weakly with the proposition that it is easy to become disoriented when using the system.

Scores by section indicate that, overall, users formed a good impression of Citebase. They found it mostly to be effective for task completion (confirming the finding of Form 1, section 2), and they were able to control the system most of the time. The lower score for navigability suggests this is an area that requires further consideration.

It should be recalled that responses were scored between 2 and -2, depending on the strength of the user's reaction. In this context it can be seen that on average no questions or sections scored negatively; six questions scored in the top quartile, and two sections just crept into the top quartile.

Among users, scores were more diverse, with the total user score varying from 31 (maximum score possible is 32) to -25. Other high scores included 29, 28 and 27 (by five users). Only eight users scored Citebase negatively.

Submission of Form 2 completed the evaluation for the user, signalled by the response shown in Appendix 2.

10 A summary view of Citebase in user comments

Feeling good about Citebase

• I am accustomed to using the ISI Web of Knowledge system which lays things out slightly differently. This works just as well once you know where everything is, though.
• The RefLinks in the PDF are great.
• The co-cited papers are a unique improvement
• The service is first rate and should be adopted by other A&I services
• Looks good - have been waiting for something like this to come along.
• Citebase is so good, I will never again use the arxiv.org search interface.

Implications for Citebase design: users

• The site is very weak in explanations on what is possible.
• There were by no means sufficient help files to assist with the simple solving of the questions posed. You are going to have to make this a lot more user friendly
• The main time delay was finding that I needed to use semicolons to separate the names of multiple authors.
• Searching multiple authors was not intuitive
• As I was looking for papers that cited a paper I used the Citation Tab on the search box, rather than the identifier tab. The "Identifier" label isn't self-explanatory.
• You need an explanatory page telling users what information is available when you click on the name of an article.
• Why I cannot see the number a paper is being cited? AND (it would be nice to have near every paper the number of citation)
• May need some training or a good guide to get the most from it.
• The interface is not as simple as researchindex.
• I'd give more prominence to the ranking options on the search screen, as they're the thing that make this database particularly useful.
• The co-citation section is a great idea, right now no one else I know has it. I think, however, that for the explanations for the Top 5, you should explain what they are the top 5 for. I am assuming they were the most cited as well
• The analysis of web server logs from the uk arxiv.org site should be removed. It has the potential to be misleading, as authors might easily raise their statistics by clicking on their paper many times.
• You need to bring in user interface specialists to improve what you have. I'm sure the underlying technology is quite impressive

Implications for Citebase design: search

• 1. A search in "all metadata fields" or fulltext might be helpful. 2. The identifier search uses exact match, it  might be too restrictive. For example, it should work if I search "hep-th/9711053"
• There might be a large number of papers on string theory which do not have "string theory" in the title or the abstract.
• Please think about strengthening the search functions, i.e. knowing that string theory finds papers that contain the word string AND the word theory is nice, but in a large database much too broad an option
• Ideally it would have a controlled vocabulary so that every article on a particular topic would have the same subject
• Many of the results that showed up were unexpected/wrong, such as papers which did not have the desired author as an author
• Your engine still has many bugs so far (concerning coverage of e-prints from arXiv.org at least): all 'ff' ligatures are deleted by your harvest engine (e.g. J. Differential Geom. becomes J. Dierential Geom.); fails to extract properly the bibliography
• The citebase search page offers erroneous references. (e.g. The Search by VALIÑA, Mª DOLORES in COGPRINTS found 1-8 papers. That is o.k., but the order of authors is erroneous in 3 papers and GLORIA SEOANE, co-author, is disappeared.

Should we be doing this?

• I am afraid that this service encourages the obsession with citations as opposed to the more reasonable research method in which one writes and reads papers for their interest and value and not necessarily for the citation value.
• The impact on research activity of the existence of such services is not necessarily benign, and will include things that are distinctly harmful. An attempt to reduce seniority rankings to numbers will have concomitant results on the quality of the research

11 Correspondence arising from the evaluation

• Citebase citation impact summary appears to give a significantly lower total number of cites than the SLAC Spires database for high-energy physics papers (links to both citation services are adjacent in the arXiv records, so comparison is simple)
• SLAC Spires indexes both arXiv articles and journals. This gives it greater coverage of all the literature in high-energy physics. Citebase can only discover citations from articles deposited in arXiv, but does so for all subject areas. We try to draw user's, and author's, attention to the warnings given on the site (http://citebase.eprints.org/help/#impactwarning). Citebase is based only on those papers, references and hit data that are available from the source eprint archives (see coverage <http://citebase.eprints.org/help/coverage.php>). These are automated processes that fail in certain conditions - for example, should an article not have a "journal-ref", or if the full-text document cannot be converted and/or parsed.
• Citebase gets the order of authors wrong
• Citebase should be displaying authors in the order given by the source repository. Regrettably the harvest program wasn't storing this order. This should be fixed once we finish our evaluation, as I will be performing a complete re-harvest (to go to OAI 2.0).

12 How Citebase has responded

Changes affecting user interface

• Updating daily
• Performance improvements (from ~3 seconds down to ~1.5 seconds for free-text searches)

• OAI 2.0 export and improved OAI export performance
• Enhanced OAI-compliant export including citation data based on the Academic Metadata Format (Krichel and Warner 2001)

• Reference-parsing fixes
• Author display-order fixes
• Reference link icons
• Identifier-tag navigation
• Treating hyphen correctly
• OAI export caching

Forthcoming developments

13 Conclusions and recommendations

The exercise to evaluate Citebase had a clear scope and objectives. Within the scope of its primary components, the search interface and services available from a Citebase record, it was found Citebase can be used simply and reliably for resource discovery. It was shown tasks can be accomplished efficiently with Citebase regardless of the background of the user.

The principle of citation searching of open access archives has been demonstrated and need not be restricted to current users.

More data need to be collected and the process refined before it is as reliable for measuring impact. As part of this process users should be encouraged to use Citebase to compare the evaluative rankings it yields with other forms of ranking.

Citebase is a useful service that compares favourably with other bibliographic services, although it needs to do more to integrate with some of these services if it is to become the primary choice for users.

The linked PDFs are unlikely to be as useful to users as the main features of Citebase. Among physicists, linked PDFs will be little used, but the approach might find wider use in other disciplines where PDF is used more commonly.

Although the majority of users were able to complete a task involving all the major features of Citebase, user satisfaction appeared to be markedly lower when users were invited to assess navigability than for other features of Citebase.

Perhaps one of the most important findings of the evaluation is that Citebase needs to be strengthened considerably in terms of the help and support documentation it offers to users.

Development of Citebase will continue, and since the evaluation has focussed on re-starting daily updates and improving performance and speed of search responses. In the longer-term, coverage will be expanded and support documentation strengthened.

A notable success of the evaluation has been to increase usage of Citebase, in terms of average daily visits, by more than a factor of 10. There is still considerable scope to increase usage of Citebase by arXiv physicists.

According to Paul Ginsparg, founder of arXiv: "(Citebase) is a potentially critical component of scholarly information architecture". The first step must be to examine the results of this evaluation to improve the services Citebase offers with a view to establishing Citebase as a service used regularly by all arXiv users.

There are wider objectives and aspirations for developing Citebase. The overarching purpose is to help increase the open-access literature. Where there are gaps in the literature - and there are very large gaps in the open-access literature currently - Citebase will motivate authors to accelerate the rate at which these gaps are filled.

Acknowledgements

Our local evaluators at Southampton University gave us confidence the evaluation was ready to be tackled externally. We want to thank Iain Peddie, Shams Bin Tariq, David Crooks, Jonathan Parry (Physics Dept.) and Muan H. Ng, Chris Bailey, Jing Zhou, Norliza Mohamad Zaini, Hock K.V. Tan and Simon Kampa.(IAM Dept.).

Finally, we thank all our Web evaluators, who must remain anonymous, but this in no way diminishes their vital contribution.

References

Bergmark, D. and Lagoze, C. (2001) "An Architecture for Automatic Reference Linking". Cornell University Technical Report, TR2001-1842, presented at the 5th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), Darmstadt, September
http://www.cs.cornell.edu/cdlrg/Reference%20Linking/tr1842.ps

Bollen, Johan and Rick Luce (2002) "Evaluation of Digital Library Impact and User Communities by Analysis of Usage Patterns". D-Lib Magazine, Vol. 8, No. 6, June
http://www.dlib.org/dlib/june02/bollen/06bollen.html

Chen, C. and Carr, L. (1999) "Trailblazing the literature of hypertext: An author co-citation analysis (1989-1998)". Proceedings of the 10th ACM Conference on Hypertext (Hypertext '99), Darmstadt, February
http://www.ecs.soton.ac.uk/~lac/ht99.pdf

Crow, R. (2002) "The Case for Institutional Repositories: A SPARC Position Paper". Scholarly Publishing & Academic Resources Coalition, Washington, D.C., July
http://www.arl.org/sparc/IR/ir.html

Darmoni, Stefan J., et al. (2002) Reading factor: a new bibliometric criterion for managing digital libraries. Journal of the Medical Library Association, Vol. 90, No. 3, July
http://www.pubmedcentral.gov/picrender.fcgi?action=stream&blobtype=pdf&artid=116406

Garfield, Eugene (1994) "The Concept of Citation Indexing: A Unique and Innovative Tool for Navigating the Research Literature". Current Contents, January 3rd
http://www.isinet.com/isi/hot/essays/citationindexing/1.html

Guédon, Jean-Claude (2001) "In Oldenburg's Long Shadow: Librarians, Research Scientists, Publishers, and the Control of Scientific Publishing". ARL Proceedings, 138th Membership Meeting, Creating the Digital Future, Toronto, May
http://www.arl.org/arl/proceedings/138/guedon.html

Gunn, Holly (2002) "Web-based Surveys: Changing the Survey Process". First Monday, Vol. 7, No. 12, December
http://firstmonday.org/issues/issue7_12/gunn/index.html

Gutteridge, Christopher (2002) "GNU EPrints 2 Overview". Author eprint, Dept. of Electronics and Computer Science, Southampton University, October, and in Proceedings 11th Panhellenic Academic Libraries Conference, Larissa, Greece, November
http://eprints.ecs.soton.ac.uk/archive/00006840/

Harnad, S. (2002) "UK Research Assessment Exercise (RAE) review". American Scientist September98-Forum, 28th October
http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2325.html

Harnad, S. (2001) "Why I think research access, impact and assessment are linked". Times Higher Education Supplement, Vol. 1487, 18 May, p. 16
http://www.cogsci.soton.ac.uk/~harnad/Tp/thes1.html (extended version)

Hitchcock, Steve, Donna Bergmark, Tim Brody, Christopher Gutteridge, Les Carr, Wendy Hall, Carl Lagoze, Stevan Harnad (2002) "Open Citation Linking: The Way Forward". D-Lib Magazine, Vol. 8, No. 10, October
http://www.dlib.org/dlib/october02/hitchcock/10hitchcock.html

Hitchcock, S. et al. (2000) "Developing Services for Open Eprint Archives: Globalisation, Integration and the Impact of Links". Proceedings of the Fifth ACM Conference on Digital Libraries, June (ACM: New York), pp. 143-151
http://opcit.eprints.org/dl00/dl00.html

Hitchcock, S. et al. (1998) "Linking Electronic Journals: Lessons from the Open Journal Project". D-Lib Magazine, December
http://www.dlib.org/dlib/december98/12hitchcock.html

Hunt, C. (2001) "Archive User Survey". Final year project, ECS Dept, University of Southampton, May
http://www.eprints.org/results/

Krichel, T. and Warner, S. (2001) "A metadata framework to support scholarly communication". International Conference on Dublin Core and Metadata Applications 2001, Tokyo, October
http://openlib.org/home/krichel/papers/kanda.html

Lawrence, Steve (2001) "Free Online Availability Substantially Increases a Paper's Impact". Nature Web Debate on e-access, May
http://www.nature.com/nature/debates/e-access/Articles/lawrence.html

Lawrence, S., Giles, C. L. and Bollacker, K. (1999) "Digital Libraries and Autonomous Citation Indexing". IEEE Computer, Vol. 32, No. 6, 67-71
http://www.neci.nj.nec.com/~lawrence/papers/aci-computer98/

Lynch, Clifford A. (2001) "Metadata Harvesting and the Open Archives Initiative". ARL Bimonthly Report, No. 217, August
http://www.arl.org/newsltr/217/mhp.html

Merton, Robert (1979) "Foreward". In Garfield, Eugene Citation Indexing: Its Theory and Application in Science, Technology, and Humanities (New York: Wiley), pp. v-ix http://www.garfield.library.upenn.edu/cifwd.html

Nielsen, Jakob (2000) "Why You Only Need to Test With 5 Users". Alertbox, March 19th
http://www.useit.com/alertbox/20000319.html

Small (1973) "Co-citation in the Scientific literature: A New Measure of the Relationship Between Two Documents". Journal of the American Society for Information Science, Vol. 24, No. 4, July-August;
reprinted in Current Contents, No. 7, February 13th, 1974,
http://www.garfield.library.upenn.edu/essays/v2p028y1974-76.pdf

Suber, Peter (2002) "Larger FOS ramifications". FOS-Forum list server, 2nd July
http://www.topica.com/lists/fos-forum/read/message.html?mid=904724922&sort=d&start=240

Van de Sompel, H. and Lagoze, C. (2002) "Notes from the Interoperability Front: A Progress Report from the Open Archives Initiative". 6th European Conference on Research and Advanced Technology for Digital Libraries (ECDL), Rome, September
http://lib-www.lanl.gov/%7Eherbertv/papers/ecdl-submitted-draft.pdf

Young, Jeffrey R. (2002) "Superarchives' Could Hold All Scholarly Output". Chronicle of Higher Education, July 5th
http://chronicle.com/free/v48/i43/43a02901.htm

Appendices

Appendix 1: Response to users on submission of Form 1

Appendix 2: Response to users on submission of Form 2

Appendix 3: Correspondence with users arising from the evaluation

Response sent to evaluator, Tim Brody: I believe SLAC/SPIRES index both arXiv-articles and the tables of contents of journals. This gives them greater coverage of all the literature in High Energy Physics (whereas Citebase can only discover citations from articles deposited in arXiv, but does so for all subject areas).

In addition SLAC/SPIRES may be able to parse references more successfully than Citebase - although we strive to improve!

Hopefully, we will be able to integrate SLAC/SPIRES and Citebase at some point to provide greater coverage (and accuracy) to the end-user/authors.

If you have specific examples of articles where Citebase's coverage is poor, please email me with URL's.

Guido Burkard <unibas.ch> (28th August):
I've recently discovered CiteBase and I like it very much!

One problem I have just noticed is that it happens that CiteBase gets the order of authors wrong, e.g. in     cond-mat/9808026 where the author listed first on CiteBase is actually the last author.

Another suggestion would be to indicate the number of citing articles on the main page--I imagine that many people are interested in the impact of their paper and don't necessarily want to browse the list of citing articles.

Tim Brody (28th August): Citebase should be displaying authors in the order given by the source repository. Regrettably the harvest program wasn't storing this order. This should be fixed once we finish our evaluation, as I will be performing a complete re-harvest (to go to OAI 2.0) - in the next month or so.

The impact figure you mention should also make its way into the abstract page, along with the hits count, and by-author values.

"Vivian Incera" <fredonia.edu> (12th September):
I am very disappointed, to say the least, with the new CiteBase system appearing in the archives. I think that it should not have been made available to users until it was bug-free. Problems as obvious as a contradiction between the list of articles citing a paper appearing in the same archive and the list you provide in your system show that CiteBase is still highly inaccurate.

I checked several of my papers and found problems in most of them. For example, a recent paper that has three citations if you click in the archive link, has no citation in your system. That's outrageous.

You know that citation is a serious matter. It is used for many different purposes, it can be used to track the literature, as well as to decide a position, etc. You can hurt many people by making mistakes like these. The idea of CiteBase is great, no doubt. I know of many papers that had cited mine, but for different reasons, as for instance that they are in cond-mat, but my paper is an hep-ph, these citations have never been automatically counted on my high energy archive list. Your system could really help to improve that situation. So, don't think I am against what you are doing. What I don't like is that the program to produce the data is still  limited, but, despite that, it is available to everybody in the web, so people can take it seriously.

I hope that you will do something to address the problems I am mentioning here. The best is to put a clear sign that this system is still under construction and should not be considered as a reliable source of information yet.

Internal project correspondence

Stevan Harnad (12th September): I agree with this user that your cautions and impact health warnings need to be strengthened to take all these limitations into account. I suggest you do that as soon as possible, to prevent reactions like this from propagating. It is just a matter of itemizing the limitations, stressing that this is still experimental, should not yet be used for actual evaluation, and is meant mainly to demonstrate the POTENTIAL of online citation analysis.

I'd be happy to vet the changed text. Politically and strategically it would be best if we got these hedges optimized and online ASAP!

The other question is about whether there is anything that can be done to fix the specific within-archive omissions she mentions. (Obviously, she is looking narrowly at her own work, and you are handling the whole corpus, but every author will be looking first, and under a microscope, at how citebase treats his own work, and only secondarily at how it helps him with the work of others.

Steve Hitchcock (13th September): I think we are talking about citations, and 'archive link' means SLAC-SPIRES HEP (cited by). It would have helped if the author had been more specific about the examples. I can't identify the paper from this.

> A recent paper that has three citations if you click in the archive link, has no citation in your system. That's outrageous.

Can anyone else? However, there is a paper that has null citations in Citebase but six in Spires http://citebase.eprints.org/cgi-bin/citations?id=oai%3AarXiv%3Ahep%2Dph%2F0004113. Maybe this needs to be looked at to see why Citebase isn't collecting these citations

It is worth noting that Spires 'cited by' lists are preceded by a prominent warning.

Please do not despair if your papers fail to appear or have few or no citations:

Citebase is based only (1) on those papers, references and hit data that are available from the source eprint archives (see Coverage <http://citebase.eprints.org/help/coverage.php>), and (2) only on those references that can currently be successfully linked (~15%). The literature currently available online is still only a tiny subset of the total research literature across disciplines, although it is growing daily. (Hence the moral of this story is not that these services are intrinsically limited, but that not enough researchers have self-archived their papers yet!)

We will, of course, try to improve these notices in response to your concerns.

We are continually improving Citebase's ability to find and link citations within the literature available to us from the source e-print archives. However, these are automated processes that will fail in certain conditions - for example, should an article not have a "journal-ref", or if the full-text document can not be converted and/or parsed.

Where we are unable to link a citation, it is most valuable to us for authors/users to supply the arXiv id's of the articles that haven't been linked together. Given the arXiv id's I can build a test case to use against our systems to identify why it failed.

As you point out, citation-indexing services should not be used in isolation. SLAC/SPIRES is likely to have more comprehensive coverage of the HEP field (due to the human-effort involved), Citebase will cover all fields in arXiv, and ISI's Web of Science will provide comprehensive indexes of the journals that they index. For any service, citation counts will be intrinsically limited by the amount of literature that is included.

Ehud Schreiber:
I just completed the exercise and the site seemed promising, but when I tried to find papers citing hep-th/0205185 I got none, although SPIRES found 30 of them.

Tim Brody (24th September): During our evaluation period we haven't updated Citebase with new records. Therefore for a relatively new article, in this case June of this year, Citebase is unlikely to have picked up any articles since that cited it.

(I've duplicated the evaluation warning on the coverage page, so if you click the "incomplete" link in the warning on the abstract page you should see the updated message)

Regardless, it is possible that SPIRES - for hep-th - will always identify more citations. While Citebase can only see citations from articles that are deposited in arXiv, SPIRES covers articles that are only available on-paper.

Internal project correspondence

Steve Hitchcock (24th September): This is an interesting example. The paper was obviously being cited within days of deposit on arXiv. Citebase has records for the earliest citing papers, 21-30 on the SLAC citations list http://arxiv.org/cits/hep-th/0205185, but for some reason has not been able to extract the reference lists from any of these papers, hence obviously no citations. Is this lack of references an updating problem, or is there a technical reason why the ref. lists are not being extracted in these cases?

Tim Brody (24th September): I *guess* that the metadata is slightly ahead of the references. Given re-harvesting everything we should pick this up again (the PDFs in question look parsable).

Appendix 4: User comments

A4.1: Comments on section 1: About you

• The choices are not as complete as needed (especially 1.1)
• * Seems simple enough
• In 1.5 may need to be clearer on the difference between OAI services and ePrints. I havered over mentioning using our own ePrints server...
• Can links to the various OAI initiatives/services be included so users can go off and explore?
• Why are there no biologist or chemist options in Q.1.1? There is much more to science that those listed, especially as this century is being hailed as the Century of Biology
• As usual, I find myself an "outsider" in discussions of things that will be important to me very soon. I find there is no category for me to go into. You guys need to look beyond geekdom to think about ordinary social scientists, librarians, educationists
• Doesn't ask whether or not I have any papers in arXiv, and, if so, approximately how many (I have only one)
• I work professionally in the field of pharmacology and molecular biology, but have an interest in broader, philosophical issues.
• Physician clinician, researcher of holistic medicine, spiritual awareness and healing
• More space for fields of interest
• In addition to CogPrints I am a heavy user of the CiteSeer ResearchIndex service. The automatic citation linkage adds a lot of value to that service.
• I AM USER AND AUTHOR of COGPRINTS
• * You could also have asked if I had *submitted* my papers to the archives as opposed to just reading and browsing
• I do not maintain anymore a "preprints" section on my website, but rather link to the arxiv submissions. i believe this makes distribution of preprints much simpler to other users.

A4.2: Comments on section 2: Practical exercise

• The Co-citing question was the hardest and needed quite a bit of ferreting around. The site is very weak in explanations on what is possible. Perhaps experts would know what they were doing but for me this was very difficult.
• I used the search in 2.6b but it was unsuccessful. I ended up looking for ref[57] in the PDF document and clicking the link from there.
• Don't understand it. The words "most cited" don't appear on the results page so I end up guessing
• Lack of familiarity with the structure of Citebase made this difficult in a hurry and I felt very unconfident - but unwilling to waste time making myself more confident.
• I think I lot of people were trying this at the same time so that complete pages did not appear. This led to much puzzlement as I did not understand where one could get the citing information, until the full page came up
• Not easy. Help documentation failed to help me in most cases. Although I understand in general the value of these citation analyses, it is not something that I do in practice. Therefore, this set of exercises was difficult to follow.
• Your questions too easily lead to the answers.  That is, you use the terminology that is used on the site in your questions. This is too leading and thus poor experimental design.
• You need clear definitions of your terms in standard English. 'Author hits', and 'Paper Hits' have no meaning for me. I'm not sure what you mean by hits, this should be made quite clear to all users. Part of the time taken was used to scratch my head
• Had problems downloading pdf for 2.6a but maybe it's my PC as we've had some updates done to it recently and there have been a few bugs. Spent about ten minutes with a frozen computer.
• Fascinating ability to cross-check links between papers, though I would need more practice to not be confused as to where I was! It seems very easy to wander off down a track (very useful and interesting in its own right), but forget where you started from
• I missed that I was supposed to time it. Other comments - interesting, but then I don't know if I interpreted the categories correctly and therefore got the correct answers.
• Not bad. I didn't even need to use any explanations. I did have a little problem with the articles citing the first article, but I think I got it right.
• Not easy!
• There were by no means sufficient help files to assist with the simple solving of the questions posed. I got frustrated trying to work out what I was supposed to be doing and lost interest. You are going to have to make this a lot more user friendly
• It took a while to figure out how to use the interface, but after I had it was fairly straightforward.
• I'm not familiar with this citation business, didn't understand Q 2.4 & 2.5
• Would have been slightly quicker to complete if the subjects of the questions came in the same order as the citations themselves e.g. asking about co-cited papers after top 5 citing articles.
• Had difficulty with 2.4 and 2.5 -- could be I wasn't clear on how to actually complete them with Citebase
• These questions start to depend on success in understanding earlier questions - and are therefore not good discriminators. I became impatient with the exercise, partly because such questions are of little interest in the fields where I have done research
• 1. A search in "all metadata fields" or fulltext might be helpful. 2. The identifier search uses exact match, it might be too restrictive. For example, it should work if I search "hep-th/9711053"
• The main time delay was finding that I needed to use semicolons to separate the names of multiple authors.
• This is an interesting test.  I must have in haste gone astray someplace as I failed to be led to the Strominger and Witten ref.
• I had some early confusion finding the citing articles and co-cited articles. I am accustomed to using the ISI Web of Knowledge system which lays things out slightly differently. This works just as well once you know where everything is, though.
• This in not intuitively obvious.
• The thing that caused most delay was getting the answer to 2.4. As I was looking for papers that cited a paper I used the Citation Tab on the search box, rather than the identifier tab. The "Identifier" label isn't self-explanatory.
• I didn't know how to do 2.4 and 2.5, and I didn't really understand the question in 2.6c. For that, I just opened the reference at the end of the article.
• The list downloads very slowly the first time. Searching multiple authors was not intuitive (I tried about 4 times before getting it right, and I had to consult the explanation link).
• 2.6c used search - but failed, so went back to the downloaded paper
• Couldn't figure out 2.4
• Exercise is ok, although I did not want to download the pdf using a dial-up connection.
• My network was disrupted in the middle and I had to start up again
• * I could not go beyond question 2.3 in spite of entering the oai identifier.
• A bit annoying, don't you think...?
• It took a while to figure out how to find the answer to 2.4; that part wasn't intuitive
• It is really easy to get use to it and seems to work well. The search engine is pretty fast.
• A pity that the references weren't extracted from the eprint. The RefLinks in the PDF are great.
• I think it would be more accurate if the search engine would accept "string theory". In the current case, it seems that some papers irrelevant to string theory are retrieved, because they are relevant either to string or to theory.
• Question 2.5 is a bit ambiguous - are you referring to the title in 2.4 or 2.3?
• I wasn't sure how to find the answer to 2.2, checked the help section but must have missed something.
• Easy to do & that usually means well thought out.
• Extremely informative, a good way to get people to learn about the handy features of the system.
• Didn't find the title of the referenced paper
• Sadly, I wasn't able to download the pdf file of the article I found.
• * In 2.6a I gave the listed link; the actual link of the download was http://ernie.ecs.soton.ac.uk/opcit/cgi-bin/pdf?id=oai%3AarXiv%3Ahep%2Dth%2F9711200
• Is impressive and I will try in my own subject during the coming time.
• * Gave me a good impression of features - in particular the co-cited papers are a unique improvement.
• * Found citebase search sort of difficult or inaccurate
• * For question 2.6d, there was a link on the citation in the pdf file, but it didn't work (maybe a configuration error on my part).  I manually typed in the url of the abstract of the reference. Also, the reference list of the paper didn't show up.
• * Useful. You need an explanatory page however telling users what information is available when you click on the name of an article.
• * I stopped after 2.3 because I found the web forms difficult to navigate. I suggest that, on top of giving forms to perform searches beginning with an identifier, you put buttons for each of the above next to the citations.
• * The "co-citing" concept was new to me and potentially useful.
• Very instructive. The service is first rate and should be adopted by other A&I services
• Nice
• Useful
• I got lost at first on 2.4. i thought i was suppose to do another search. i finally found the answer on the full display of the previous article.
• Nice exercise
• * It was the first time I tried these services. So hope that I will usually be faster.
• Very good for teaching the capabilities of the search engine. I just wish I were less tired.
• * These exercises were very helpful to me. I am not sure whether my answers are correct because there might be a large number of papers on string theory which do not have "string theory" in the title or the abstract.
• hmmm... there must be a difference between "rank by cocitedness" and the "cited with" button, but i can't guess what it is.
• Why I cannot see the number a paper is being cited?
• * It is a bit confusing...
• * (it would be nice to have near every paper the number of citation)
• I could not do 2.6 b, I did not get a complete list of references from the link "All Articles Cited by this Article (Reference List)''.
• * I can't be bothered doing this.
• * It showed what other kinds of things one can search besides the most cited author or paper.
• * This appears faintly unethical. To my way of thinking the whole point of the archives is to allow open, swift exchange of ideas in the form of manuscripts.  By making "impact" tabulations available the archives are diluting their purpose
• bah, I can't deal with this nonsense yet...keep working at the interface, and/or hope for a better class of guinea pigs!
• uninteresting, did not find motivation to complete
• * It was much easier to download the paper than to find the URL. This is probably not the question you wanted to answer.
• Author order is NOT correct. Order in citebase appears to be alphabetical.
• * Of course, one can find the answer to 2.6.b in the article: Black Hole Entropy in M-Theory, but see error-message above.
• * I know very little about string theory
• * I do not care about strings, but it works with other keywords
• * Excellent!!! I would appreciate, thought, if I could email people something like a business card - something like: http://citebase.eprints.org/cgi-bin/search?Rui+Carvalho - that would send them to a page where all my preprints would appear

A4.3: Comments on section 3: views on Citebase

• I ran into two problems doing the task: (1) it wasn't clear at all initially how to find the most cited author (to do that task you'd have to go into the dropdown) -- I ended up doing the tasks backward and managed to find that feature in the dropdown when I was doing something else. (2) it was a little hard to find the paper given three authors -- I couldn't find the paper searching by author or by it's hep-th number and ended up having to look it up in arxiv.org.
• Pleasant layout
• Favourable, we'll use it next time provided that more explicit explanations are offered
• Pretty good with features on cocitation etc.
• * Citebase isn't quite as complete or comprehensive as, say, INSPEC, at this point.
• I chose not to respond to 3.1 because it is not possible for me to choose among these. They all seem useful to me.
• You need to bring in user interface specialists to improve what you have. I'm sure the underlying technology is quite impressive
• If it was clear (to me) what the database was doing then it would be quite useful for searching, and finding seminal papers in a field. Ideally it would have a controlled vocabulary so that every article on a particular topic would have the same subject
• The search options don't seem particularly powerful and I can imagine there being problems finding some things within these limits.  However, overall it is an excellent idea.
• Very clever and valuable resource. May need some training or a good guide to get the most from it.
• If this project goes forward, please think about strengthening the search functions, i.e. knowing that string theory finds papers that contain the word string AND the word theory is nice, but in a large database much too broad an option
• Probably not much use in my area - ornithology, bibliographic research pre-computer days
• With better explanations and an improved interface this would be an excellent service.
• Great to have this facility for eprints - might make people more happy to use them which would be a good thing!
• Looks great -- would love to see something similar in the human sciences
• Its purposes are unfamiliar to me. I am interested in subject matter, methodological approach and scale, but not with citations.
• 1. After search, the number in front of matched articles is a bit confusing, it has different meanings based on ranking criteria. 2. I used researchindex.com a lot. The interface is not as simple as researchindex. After all, the critical one is number of
• A very interesting service!
• It's a good idea. Since it is so slow, I shall probably use it only for special situations.
• The citebase search page offers erroneous references. (e.g. The Search by VALIÑA, Mª DOLORES in COGPRINTS found 1-8 papers. That is o.k., but the order of authors is erroneous in 3 papers and GLORIA SEOANE, co-author, is disappeared. See COGPRINTS: cog00001
• I took a look at this and liked the general approach.  As a music technologist, I found that there was little content of interest, however.  The boundaries of "science" are quite a big bibliographical problem. My student use PsychInfo, CiteSeer, and RILM
• Once I had got my head around question 2.4 "Which is the most highly cited paper that cites the most-cited paper above?"- (I was attempting to do this via the search screen instead of using the data attached to the citation) then I was well away!
• The good news is that the pdfs are there.
• The subject coverage is too narrow. Would need more engineering and chemistry coverage.
• A bit confusing at first (the brown Abstract/ref list etc. links looked like titles at first on the paper's screen). I'd give more prominence to the ranking options on the search screen, as they're the thing that make this database particularly useful.
• Excellent. It would appreciate if it would includes more papers and other fields (logic, philosophy)
• The Co-citation section is a great idea, right now no one else I know has it. I think, however, that for the explanations for the Top 5, you should explain what they are the top 5 for. I am assuming they were the most cited as well, but I that info wasn't
• The most important improvement is online access to the full text. PubMed provides full text only for some journals.
• Citebase needs OpenURL support
• * Papers download with filename "pdf" only
• * The coverage is still too small to give me a reliable overview, especially in fields bordering and including engineering. But the features that are implemented have impressed me tremendously.
• * Many of the results that showed up were unexpected/wrong, such as papers which did not have the desired author as an author
• * Looks good - have been waiting for something like this to come along.
• * I would have to play with it a fair bit before I could comment intelligently about it.  It seems, however, a good idea.
• * Have to use it more to comment.
• * Archived preprints are usually published somewhere, citations to the preprint and the final published version should be combined in Citebase.
• * The analysis of web server logs from the uk arxiv.org site should be removed. It has the potential to be misleading, as authors might easily raise their statistics by clicking on their paper many times.
• Very clear and succinct search processes
• * So far used my own bib list, but would like to try this out.
• * I am afraid that this service encourages the obsession with citations as opposed to the more reasonable research method in which one writes and reads papers for their interest and value and not necessarily for the citation value.
• * Citebase is so good, I will never again use the arxiv.org search interface.
• Well, most of the papers i refer to are in (printed) journals, so if there's no search possibility for these it seems like i miss 90% of what i'm looking for. I'm not really convinced that it's useful to know how many people looked up a paper; there'
• * Hits/month feature useful, but is it up to date?
• * Way to go!
• * It is as bad as all other citation databases; I don't see the reason for setting up such a database
• Your engine still has many bugs so far (concerning coverage of e-prints from arXiv.org at least): all 'ff' ligatures are deleted by your harvest engine (e.g. J. Differential Geom. becomes J. Dierential Geom.)
• Data base too small. Should contain all other OAi data providers containing physics, PhysDoc, etc.
• I do not think this database is a good idea in general. It only lists papers in or related to ArXiv which is a very much incomplete source of information in mathematics. I think using it in any way to evaluate someone's work will be misleading
• Very good indeed. hope it remains free
• * great initiative to break the monopoly of ISI
• * very noisy results, missing articles. No match today of services like inspec or science citation index. The idea of course is very good, and I hope the quality improves. I would also suggest one installs an icon to retrieve information in bibtex format.

A4.4: Final comments on the evaluation

• The questions are quite comprehensive & concise.
• Good idea but led to confusion because of too many people trying it at one time
• Well-thought out interactive test, although I'm itching to check whether I found the right articles now!
• Too long an exercise and questionnaire for casual testers
• A waste of my time.
• The impact on research activity of the existence of such services is not necessarily benign, and will include things that are distinctly harmful. An attempt to reduce seniority rankings to numbers will have concomitant results on the quality of the research
• Required about 25-30 minutes total
• You don't really expect people to do the practical exercise do you?
• This was very interesting. I look forward to seeing functionality like this on the Cogprints archive.
• Nice way to create a remote evaluation
• took too long, slow loading, incomplete explanations of use
• I really look forward to see your final version.
• very convincing
• this is really well done as an evaluation. I am impressed.
• precise
• I think the evaluation should be asked after some time using Citebase.