Open Journal project: final report to eLib

 

by the  
Open Journal Project  
team

 Projects funded by the Electronic Libraries (eLib) programme in the UK are required to submit annual reports of progress and are encouraged to post a version of the report to the Web. This report covers the third and final year of the Open Journal project from May 1997 to May 1998, and also reviews the achievements, results and lessons from the project's full three year period from 1995. It is intended that this report can be read alone as a record of the project's work, although some technical details are referred to the previous two annual reports covering the first year and second year of the project. . 

Submitted to eLib: August 1998 
This edited version posted to the Web: November 1998
References updated 15 December 1998

Contact for this report 
Steve Hitchcock 

Open Journal Project
Email: sh94r@ecs.soton.ac.uk 

Contents

Part 1. Review of final year (year 3, 1997-98)
 

Part 2. Overall project review (1995-98)

 

Appendices

Contacts

Review of final year (year 3, 1997-98)

1  Year 3 Introduction

Part 1 of this report covers the third and final year of the Open Journal project from May 1997 to May 1998.

Highlights of year 3


Summary of the year

A feature of the project's final year were the initial moves towards publisher-specific projects, as anticipated in the original project proposal but effectively recasting the publishing framework within which the project had worked, and consequently the reduced emphasis on the development of supporting technology.

Agreement was reached with the contributing publishers to make one of the Open Journals developed in the project, in cognitive science, openly accessible for a limited period. Previously, user tests had been restricted to selected, subject-specialist users. This large-scale test principally demonstrated citation linking, suggesting this will be a dominating feature of future e-journals and producing valuable findings with regard to usability, quality, reliability, flexibility and efficiency. The first tentative moves elsewhere among publishers towards citation linking services suggest these are emerging issues that have not yet been confronted.

As part of an informal continuation strategy, a number of publisher partners have begun to investigate link publishing strategies specifically for their own resources. The most direct manifestations of these developments to date were installing the link service suite of software on one publisher's server, at the Company of Biologists, and the addition of another new partner, BIDS, to the project with a particular remit beyond the project timeframe. It has been notable how different each publisher's requirements have been for linking applications.

This was one difficulty in forming an exit strategy, which can also be attributed to other factors, such as the need for wider publisher collaboration, better tools and greater user awareness, some of which may be discounted in time. For this reason some of the development work and partnerships established during the project are being sustained beyond it.

As a result of these publisher-led developments, work on supporting technology was directed less at the broader innovations that had been anticipated previously, and more towards refinements that would assist application of the tools by publishers' Webmasters and which, informed by the project's results, would support requirements newly identified by publishers. The ability to serve static links in PDF files - typically the link service applies links dynamically, or 'on-the-fly' - is an example of such a publisher-driven modification.

Other planned technical developments had been predicated on extending the existing Open Journals but were altered due to these publisher interventions. The prospective publisher applications do not seek to develop the Open Journals built with the information resources collected in the project. As this became apparent, work on the two Open Journals in biology and computer science concentrated on supporting the individual publisher models rather than the broader scope envisaged.

Details and results of the project were widely disseminated during the final months with presentations and workshops at national and international meetings, published papers as well. A seminar was held for project partners.

Report against proposed year 3 deliverables

The following deliverables, accompanied here with brief comments, were cited in the project's original proposal to FIGIT.

2  Year 3 Activities and Progress

Summary of previous years

To put the year's activities in context, here is a brief summary of developments in Open Journals and user tests described in previous annual reports. For a complete list of publisher partners and people involved in the project, see Appendices 2 and 3.


Technology: the interface between users, documents and links

The link service: enhancements and application in year 3

The major link service developments for journal applications (e.g. PDF linking for the Open Journal of Biology, and bibliography linking for the Open Journal of Cognitive Science) were pioneered in the second year of the project. The third year involved integration of these other software components with the link service, and produced refinements in response to earlier user feedback, bug fixes, and customisation for some of the publishers involved in the project.

Integration of software products. Two key issues identified in previous reports were the problem of maintaining, upgrading and adding new functions to the suite of software programs used in the project, and the need for faster processing to deliver linked documents to users. One way of tackling both issues was to redesign the different programs to work more closely through tighter integration with the link service, and this became a priority of the technical work in the final year, with implications particularly for the PDF linking tools and citation agent, described below.

User feedback: an explicit proxy. The link service required users to adjust the proxy setting manually  in their browser. For many this was inconvenient; for users in libraries or on corporate networks it was often not permitted, for security reasons. As a result the link service was customised to allow the proxy to be set explicitly as part of the URL, so that the links on every Web page it served were automatically rewritten to allow those linked items to be delivered via the proxy in turn. This approach was tested in the latest release of the Open Journal of Cognitive Science.

Debugging an OJ: looking for 'wormholes'. The native versions of both primary journals used in the Open Journal of Cognitive Science, that is, prior to application of the link service, had navigation 'wrappers' which contained links to many different information resources and services. To apply an explicit proxy, as well as adding new links such as citation links, the link service had to rewrite all of these native links, which would otherwise act as 'escape routes' to resources not included in the Open Journal. One of the preoccupations in finalising the Open Journal was to eliminate as many of the 'escape routes' as possible so that the link service, acting as an explicit proxy, never lost track of the users' sessions through HTML forms, CGI scripts, and the like.

Customisation for publishers. The model that Web publishers are used to is of maintaining a core of data which is used to generate a static HTML Web site. Consequently the link service approach can run into a psychological barrier with publishers. Late in the third year the link service was altered to allow it to be run as a batch process to create a new, offline and static 'linked archive' for the Company of Biologists. The company intends to use this version of the software to add links that have been created by its editorial assistants. The role of the link service is thus focussed on manipulating the PDF files, whereas delivery of the documents is left to the standard HTTP daemons and creation, management and storage of the links themselves is left to bespoke webmaster-specified applications. This fits well with the Web publishing model, reducing the barrier to take-up of the link service software.

Link behaviours. Extra link display facilities were added to the link service. These facilities (including the use of dynamic HTML for creating link menus and lists of pending links) were never released as part of an Open Journal because they depended too much on specific browser features, although they were displayed at the seventh international World Wide Web Conference in Australia. A further link variation, trails, was implemented as part of ongoing involvement with the JISC-funded Nature digitisation project.

Relevant papers: [6] [7] [8]
 

Agent technology

The citation linking agent, first demonstrated supporting the Open Journal of Cognitive Science during year 2, was developed further this year. Enhancements to the agent technology were designed to generalise its application to various journal citation styles, to increase processing speed, to improve integration with the link service, and to work with the PDF tools.

Recognising different citation styles. There are two or three broadly accepted styles of referencing used in scientific journals, but within these styles there can be an almost endless number of subtle variants which are often not just publisher-specific but journal-specific and evolve from a journal's 'house style'. While an automated agent used to recognise citations depends on the correctness and consistency of references, it can have built-in rules, or heuristics, to identify specific styles. The heuristics, in the form of sets of regular expressions, recognise particular patterns in the document text as playing particular roles in a citation or bibliography record. One of the aims of the third year was to make the application of these patterns sufficiently adaptable to work across multiple journals. A collection of heuristics applying to a particular journal might be called a filter. The agent analyses the first citation to work out its format and then selects an appropriate filter to apply to the subsequent citations to extract the data. Initially based on rules for the journal Psycoloquy, filters were added to enable the Open Journal agent to work with Behavioral and Brain Sciences (both in the Open Journal of Cognitive Science), and other journals included in the Open Journal of Computer Science. Although the two journals above were the main targets, some work was done in applying these techniques for less constrained situations, e.g. journals in the BIDS Journals Online service and also for the Cognitive Science grey archive, the eLib Cogprints project, with sufficiently promising results to justify further investigation.

Improving processing speed. Since citation linking is performed in 'real time', initiated by a user request for a document and performed during download, speed is a critical factor in this intensive data process. The speed of the agent was improved by:

Pre-processing page links. Another provision to improve speed was modifying the agent to be run offline to provide pre-processed versions of known pages. In this case users do not have to wait for the agent to filter a paper if it is found in a cache of pre-processed papers.

Integration with other software tools. Conceived as an autonomous processor, the agent nevertheless has to work other software processes in journal linking applications. The agent can communicate as an external process, but this is not always the most efficient implementation where the other processes are known in advance. Partitioning the agent as a 'library' of functions allows it to be used by other programs without running it externally. In the project this approach was used to enable the agent to be used with the link service and PDF software.
 
Relevant papers: [1]


PDF linking: enhancements and developments in year 3

To support maintenance and upgrades and improve speed of performance, integration of the PDF-handling software with the link service became the priority of this aspect of the work in the final year. As a result the strategy of producing a separate PDF toolkit was replaced by building a PDF module within the link service. The PDF module was extended over this period to support much of the linking functionality supported by the HTML modules in the link service, especially citation linking for which a CGI resolver script was written.

Citation linking in PDF. By the end of the project's second year, the PDF toolkit had enabled keyword links to be incorporated into PDF files by extracting words and phrases from the PDF. This process used the generic link service technology to access linkbases and to determine links that could be applied to a document. This functionality was included in the PDF module, and was extended to recognise and link citations (in this context, 'citations' refers to the bibliography or list of references typically occurring at the end of an academic paper, not the citation occurring in the text).

To perform citation linking, a PDF article requested by a user is scanned until a likely phrase delimiting the start of the reference section is encountered. Typically this will be the word 'Reference' or 'Bibliography'. Further checks (point size of the text, whether there are other words with the same baseline, etc.) ensure this phrase is not part of the running text of the article. When the reference section has been determined an attempt is made to identify each individual citation. Regular expression matching is used to determine the citation label (typically one or two digits followed by a period, or a combination of letters and digits, e.g.  12. or AHO1997a).

Once the reference has been identified an HTML string is created, enabling the minimal formatting of the reference string to be encoded. Some of this formatting (e.g. font changes to italic or bold) can provide useful clues in understanding the component parts of the reference string.

With the HTML reference string, the software can attempt to resolve the reference. There are two ways of doing this. The first is to use the citation agent developed in the project. The second is to use a simpler technique. Although the PDF module will support the citation agent this was somewhat unwieldy and the simpler citation mechanism was used in one application, the Open Journal of Computer Science.

Citation matching: a CGI resolver script. The simpler citation matching scheme involves the creation of a URL that points to a CGI script residing on the project's Web server. The PDF module incorporated in the link service parses the HTML string and generates a URL if the journal title specified in the string is recognised. If there is a match a link is created around the reference, but the destination of the link is not resolved at this time. Instead, the link included in the PDF document points to the CGI resolver script. The arguments to be passed to the resolver script are encoded in the URL, and include information such as the journal title, volume, issue and page number - all of which are extracted from the HTML string. When a link is activated the CGI script parses its arguments and looks for the particular reference. If the document is known it can be delivered to the user. If the specific document is not known the CGI resolver script points to the home page of the specified journal.

On application in the Computer Science Open Journal, this simpler method of citation matching did not work as well as originally hoped. The recognition of citations and the link inclusion worked, but generating the destination URLs to specific documents via the resolver caused problems.  The net result is that while citation links are displayed in a PDF file, following those links frequently fails to produce the cited documents. Given the small size of the target database, the number of citations that can be resolved to specific documents will be limited. However, because the full citation is not resolved at the time of matching, only when the link is activated, this offers more links than are actually possible, consequently giving the appearance of reduced link reliability. One solution is to resolve the links fully at the time of matching, producing fewer links, but working links, as implemented for the Open Journal of Cognitive Science, although this demands much faster processing capabilities, especially for PDF documents.

PDF processing: new features. Other enhancements to the PDF module of the link service enable the inclusion of a 'watermark' facility and themed trails in the bookmarks section of PDF files. With the watermark a phrase such as 'This document has been enhanced by the OJF on day/month/year' can be emblazoned across the first page of the PDF either vertically, horizontally or diagonally. Trails enable a linkbase author to relate articles in a particular sequence. When a trail linkbase is included in the user's Open Journal preferences and an article contained in the trail is encountered, two bookmarks, specifying the trail title and the next and previous articles in the trail, are appended to the PDF bookmarks. Trails were first developed for use in the collaboration with Nature.

Relevant papers: [5]
 

The Open Journals - year 3 updates

Cognitive Science: version 2.1, open release

As perhaps the most successful application and demonstration of the technology, this Open Journal produced most data about the needs of further tools development and produced new perspectives on the requirements of publishing applications. The results of an important user test performed with this Open Journal are presented later in the report.

Web page design and linking. One of the features advocated for this approach to linking is its flexibility with respect to the location and ownership of linked materials. Early in the project attention on this feature concerned likely publisher reaction, and how this approach can work within current legal constraints, copyright and author's moral rights, etc. Cross-publisher agreements where this approach might work, such as announced by ISI, are emerging. A more immediate problem has appeared, however. From a user perspective, the generation of links cannot be separated from Web page design. Web design has evolved dramatically during the project, with de facto standards emerging for gateways pointing at large numbers of resources, and many simple documents now showing added features. In these schemes links become not just activators but signifiers within layouts. Feedback from the publishers and from initial user tests showed that the presentation of the bibliographic data and the facilities for navigating the database were as important as the link functionality, such as the ability to travel forwards and backwards in the database. This indicates that a better understanding is needed of how Web design and linking might converge, especially in the area of user interfaces and 'gateways' to collected resources. In most Web applications design and linking are typically separate processes.

Fixing 'wormholes'. The move towards a proxy-based link server and subsequently to an explicit proxy, or proxy-in-the-URL, has been explained above. This Open Journal was among the motivators for these changes, and in version 2.1 adopted the explicit proxy. It is vital in this arrangement, because the user cedes all control over the link proxy, to avoid any links in a document which do not include the proxy address, because these are routes out of the Open Journal, referred to as 'wormholes', from which the user is unable to return. For large resources, it is not a simple task to ensure complete coverage - every contained link must include the proxy. Web forms and local search services, for example, have to be managed carefully. Again, user testing reflected the magnitude of this task, reporting various  incidences of 'holes' which were then plugged. A more systematic approach to linking via an explicit proxy needs to be developed. This is another result that suggests a tighter integration is needed between Web journal design and link service application.

Direct citation links. Another surprising result was the preference for users to jump directly to linked materials wherever possible, rather than via some intermediate stage. For example, originally all citations in the main body of a paper in this Open Journal linked to the reference list at the foot of the paper, and then from this list linked where possible to the bibliographic (ISI) data. This approach can be modified to link in-text citations directly to the ISI data, effectively cutting out the need to look at the reference list. It is not clear that this user preference would extend to direct linking to full text were this possible, because user responses elsewhere, although not conclusive, suggest the need for a 'metadata' level of information, such as bibliographic data, for initial research. Clearly, this result might have very significant implications for the current structure of academic publishing and the inter-relationship between primary and secondary sources.

Identifying citation styles. The report has already described how the citation agent has been adapted to recognise citation styles in different journals, and it was this Open Journal that first highlighted the requirements. In most near-term applications of this sort the largest single resource is likely to be the secondary data, or bibliographic database. One of the advantages of the link service approach is that in principle this large database has to be prepared, indexed, etc., only once (and maintained!); new primary journals can be linked into this resource without changing the secondary data. This should not diminish the importance of preparing the primary data for consistency, as necessary, or preparing the agent to recognise the reference style of the new journal. After working successfully with the journal Psycoloquy, the addition of Behavioral and Brain Sciences to the Open Journal was less straightforward for the agent application. Although some journal data were re-edited and the agent rules supplemented, this part of the Open Journal still produced a higher number of link errors. The precise reasons were not clear and there was not time to investigate fully. It could be due to the fuzzy matching technique used by the agent to locate bibliographic data, with implications for design and reliability, but another possible explanation in this case may be inconsistent journal data. With a large journal archive, preparing data could be a substantial task.


Biology: migration to a publisher

The first Open Journal to be produced, in year 1 of the project, this Open Journal progressed beyond the laboratory. Real applications, if somewhat limited initially in terms of the use of the link service, are in prospect. Yet, given the resources available for linking, including gene sequence databases as well as a good collection of primary journals, this Open Journal did not realise its full scope as a demonstrator within the project. This was partly due to some early discouraging results when the technology was less mature, but also due to similar developments elsewhere, such as widespread linking to the Medline abstracting and indexing service, as described by BioMedNet [9]. Medline linking examples typically link a local copy of the Medline data rather than use distributed resources as in the project. There were also differing views among the participants about future directions for this work.

Migration to Company of Biologists. At the start of the project's third year, the Company of Biologists acquired its own in-house Web site and Webmaster, presenting an opportunity to get feedback about the installation and maintenance of the link service software in a publishing environment. Responding, COB noted that the configuration set-up should be changed to be like that of a Web server, and that the user interface should be easily configured to adapt to the host site's look and feel. This resulted in a new version of the software, hosted at the publisher's site. The Biology Open Journal demonstrated in the project was too unfocussed as it stood to be useful to the company, but a highly focussed sub-domain of the subject area - the Naemoatode worm - has been identified which fits in with the publisher's archive and development strategy. This subdomain is being used as the basis of a new Open Journal which it is hoped wil be released as an electronic-only 'special issue'.

Reduced citation linking. In a second iteration of the COB-hosted Biology Open Journal a cut-down approach was developed for use within the PDF parser so that citations in articles from specific publishers could automatically link back to the publisher's own archives. In this case citations to any COB-published journal article were linked back into its Web site. This was worthwhile. In one of the journals, Development, some 16% of citations within the last five years cite the journal itself.

Large biology linkbases require link service/PDF debugging. It was discovered that the large link databases available for biology could cause problems for the PDF linking software. Investigations showed memory requirements and processing time growing exponentially with the size of the linkbases. Further exploration of the effect found some bugs in the link service  code and some undocumented features in the Adobe PDF library. The result was that an average-sized article in Development, containing 40 references, could be processed in 1 second.
 

Computer Science: internal release

The Computer Science Open Journal was created in the last year of the project. In contrast to the other Open Journals, all of the primary journals involved were in PDF.

This Open Journal demonstrated the keyword linking facility of the link service, but there were problems with citation linking in PDF, as described above. Due to these continuing difficulties this Open Journal did not, at formal completion of the project, show any additional features that were not already evident in the Biology and Cognitive Science Open Journals so it was not released outside the development team.

The approach to citation linking in PDF worked best for self-citing (a journal paper citing another within the same journal), as shown by examples based on the journals EPODD and SPE, although these journals, which were made available to the project by Wiley, are not generally available over the Web. Many references were linked but when links to journals other than these two were activated the resolver script was unable to resolve the references exactly and in such cases returned the journal home page of the cited article. This two-stage citation resolving process, while allowing articles in PDF or HTML to be processed faster initially, did not prove successful in this Open Journal, for the reasons described above in the PDF linking section.

Keyword linking was more successful, with many links being added to articles. It was not uncommon for thousands of links to be added if an article contained many occurrences of popular words in the linkbases. Additional filtering techniques to reduce the number of links, and methods of distinguishing between links of differing priority (coloured boxes, asterisks, etc.), could be  considered for future development where it is felt this would reduce both the time taken to compile links into articles and the machine overhead in performing the link inclusion.
 
Although PDF linking has been problematic, the widespread use of PDF by journal publishers suggests that the problems are worth solving, and it is likely this development will continue in conjunction with other PDF publishing projects. The use of metadata containers for citation (and possibly keyword) linking is likely to be considered in future work. Rather than attempting to link to actual resources, links should be made to metadata containers (e.g. RDF and related technologies). Metadata containers that describe the functions (translations, structures and components) as well as standard metadata such as author, title, journal, etc., should be investigated.


Evaluation with users

The project presented a broad message and disseminated widely, as shown below in Table 1. As a result it appears that few people involved in electronic publishing or digital libraries were unaware of the project. It is more difficult to gauge, however, perceptions and expectations of this diverse audience. How much did these impressions contribute to the response when Open Journals were eventually released for user testing, because motivating users to comment was not easy. Were potential users afraid of the technical nature of the project? Ironically, much of the later progress in the project involved hiding many of the technical features from users, in response to earlier feedback.

Throughout the project various strategies were employed to attract and involve users:

Vitally, the first major user test was by personal visit, and it informed much of what followed.

In all cases building these user groups was easy. Many volunteered willingly, independently even, such was the level interest in the work. Over three years well over 100 potential users registered to be informed of releases. Subject specialists also appeared highly willing when approached. This level of interest did not translate to real responses to the Open Journals, however. The nature of the communication, and timing of releases - in some cases there was a delay in users seeing Open Journals after signing up - were no doubt contributory factors. So too were the Open Journals, and a judgement has to made about the level of response and its implications for the Open Journals. Despite this replies were generally positive and constructive, as can be judged below.
 

A major release

During the year reported here the most significant, and the last, user test was the open release of the Open Journal of Cognitive Science. This was announced by email notices to various lists: Due to the time-limited nature of this release, notices were not sent to archival sources, Web journals such as Ariadne or D-Lib.

Responses were received from over 20 users. Some lessons are drawn from this test in the section reviewing the project. The user responses and questionnaire are reproduced in full in Appendix 1.

Relevant paper: [10]
 

Publishers

It was not expected that there would be major changes to the list of publisher partners during the final year, but one new publisher did join, one left for reasons unconnected with the project, and an attempt was made to obtain permission to use data from a US publisher.

In. The new publisher to join was Bath Information and Data Services (BIDS). Discussions on BIDS' involvement started in March 1997, and it formally joined the project in March 1998 with a representative on the project steering group. Among its activities BIDS is a journals aggregator, and the aim is to apply some of the linking strategies developed in the project across the range of journals in its Journals Online service. This arrangement added no new resources to the existing Open Journals. Investigations will continue beyond the project.

Out. Chapman & Hall left the project in March 1998 after its acquisition by Wolters Kluwer. Although the project has a contact within Kluwer, at that late stage no attempt was made to resurrect the arrangement with the C&H journals as it would have involved finding a new representative from the publisher.

Prospective. The Association of Computing Machinery in the US was approached in April. An opportunity arose to apply a well-formed set of links to a new electronic journal supported by the British Computer Society and Oxford University Press, and the ACM was invited to collaborate. Past experience from the project shows that trans-Atlantic collaboration can be difficult to initiate at short notice. Since then other groups have shown interest in cooperating in this work so the particular issue remains live and will be pursued.

The list of publisher partners involved during the project is included in Appendix 2.

At the conclusion all publishers were invited to submit proposals to continue work from the project or explore new ideas - some had already done so - and these initiatives are briefly described in Publishers: the way forward as part of the overall project review below.
 

Dissemination

Open release of one Open Journal, and the freedom to demonstrate a working model publicly, gave the opportunity for wider exposure through notices, user testing, workshops and seminars, as well as continuing presentations at conferences (Table 1). As a result, the last 4-5 months of the project were particularly active for dissemination using a plan approved by the project steering committee at the end of 1997.
 
Dissemination was targetted at a broad range of interests - developers, researchers, scientists, teachers, publishers and librarians - in the UK and overseas, with presentations this year given in the USA, France and Australia.
 
 
 
Table 1. Presentations at conferences and other meetings 97-98
Type of meeting Title: Organiser Where When Principal audience Attendance Speakers from project Ref.
International conference Digital Libraries '97: ACM Philadelphia, PA, USA July 1997 Researchers, librarians c. 350 Steve Hitchcock [1]
Seminar Open Journal publishers' seminar Southampton, UK December 1998 Project partners, publishers 20 David Brailsford, Robert Kimberley, Owen Garrett,  
Les Carr		 na
One-day seminar Web Database Publishing: BCS specialist group on Electronic and Multimedia Publishing London, UK February 1998 Technical developers, commercial and research 100+ Les Carr, Andrew Witbrock (BioMedNet) na
Conference IRISS'98, Internet Research and Information for Social Scientists: ILRT, Bristol University Bristol, UK March 1998 Social science researchers, teachers c. 150 (in one session) Robert Kimberley, Steve Hitchcock [3]
Conference: workshop Annual conference: UK Serials Group UKsg Exeter, UK March 1998 Publishers, librarians, academics c. 30 (for two workshops) Steve Hitchcock, Les Carr [4]
International conference Electronic Publishing '98: IRISA St Malo, France April 1998 Technical: developers, researchers 60 Steve Probets [5]
International conference Seventh WWW Conference: IW3C2 Brisbane, Australia April 1998 Technical: Web developers, researchers 1500 Wendy Hall [6]
One-day seminar Making the most of e-journals: UKOLUG/UKsg Loughborough, UK April 1998 Librarians: academic and industrial c. 80 Steve Hitchcock [10]
International conference Hypertext '98: ACM Pittsburgh, USA June 1998 Researchers 150 Les Carr [8]
 
For details of subsequent publications, see in this report Electronic journal publishing in context. An additional strand to dissemination has been the project's presentations and publications on more general developments in e-journals publishing. These works were not explicitly about the project, but were based on research intended to inform the project of developments in the publishing environment it was targetting. Yet publishing the results of these surveys has brought international attention to the project. By fortune of timing the project's first e-journal survey in 1996 received most response. A partial follow-up published during the project's final year [2] was itself the result of an invitation to extend the original survey.


3  Learning from Experience

Clarifying the vision: impact on exit strategy. Vision is an important motivating factor for innovative work. It is important that all members of a development group understand and share the same vision. This may often appear to be the case, but can be deceptive. A primary objective of the project was to create an Open Journal 'framework', the original proposal describing this as a "framework of information retrieval technologies and electronic publishing practises to be used by information providers". That framework was to be used, among other objectives, to "develop a set of ‘open journals’ based on existing publications". Yet at stages in the project it may variously have been believed that the product would be a collection of software tools, individual software tools, a service which used these tools, or a set of link databases for distinct journals or for journals collectively - the Open Journals. This apparent lack of clarity particularly affected the discussion of the project's exit strategy, and consequently the way the final year developed. One reason for the different perceptions is that the project grew substantially - doubling the number of publishers and contributed resources - after launch and some time after the original proposal. New partners have an influence. The path to innovation is never smooth and throws up new perspectives. What has been shown in this case is that the original vision needs to be robust and strongly held by a nucleus of participants if it is to withstand such challenges, and the earlier it is tested the better.

Tackling diverse agendas: impact of exit strategy. Superficially a project involving two types of partner, developer groups and publishers, would not appear to be especially diverse. But publishers have been described as notoriously anti-Procrustean; indeed, it is seen as a strength in a competitive industry. So the project is not the first such collaboration to have encountered the problem of competing interests and hidden agendas. Recognising and tackling the problem, however, are different matters. Open communication has to be encouraged, but creating the environment in which this can happen is difficult. One way of achieving this is to place the project interests beyond the scope of competing, or commercial, interests, and a research project may have the chance to do this. For development work, however, such a strategy cannot survive through a project. There is also the demand within the eLib programme to create an exit strategy, an understandable and important objective. It has been noted in Tavistock reviews of the eLib programme that some projects found it difficult to produce business plans and continue development. There is another aspect: not just the pressure of time, but the imposition of a schedule for exit strategies. As for the Open Journal project, there is no doubt that the need to discuss an exit strategy to a programme-led rather than a project-led schedule altered the scope, momentum and complexion of the final year work. Given the structure of the project and its partners, this was a process it obviously wanted to tackle, but it may have been more productive, and might still be, had a business strategy been allowed to emerge more naturally, after more testing and further releases, because this remains an area where development and technology remain in advance of cultural and commercial requirements.

Evaluation is a good thing! Independent evaluation of a project is vital. If there is one lesson that might be applicable to other collaborative eLib projects, it is to appoint an external evaluator early. The evaluator is positioned to identify fault lines - tensions and anxieties - between collaborating groups. These lines invariably materialise over time, are often difficult to discern by the direct participants or are concealed, and become more difficult to tackle both because they can become embedded in practices and procedures and because issues of ownership, of ideas as much as products, become more acute later in a project. Such tensions may concern major or minor issues, but will always have an effect on the progress and efficiency with which the work is prosecuted.

In the case of the Open Journal project, the evaluator's report is included in part 2, the overall project review.
 
 
 
 
 

4  Year 3 Project Management

Finance

This section of the report detailed the project finance and spending, and was included in the version submitted to eLib and circulated to project partners. We are unable to publish the data here.
 

Steering group

The project steering group, which directed the project, met five times during the final year, 15 times overall. The steering group included representatives from all major partners. During the year some partners brought more members into the group as the role of meetings took on a stronger reporting function. Attendance was invariably good, between 10 to 15 people, typically more than 75% of eligible members, as it had been in earlier years, reflecting a continuing commitment to the project by most partners. Only once did attendance fall significantly below this level during the whole project. The external project evaluator was at a number of the meetings this year.

Meetings alternated between Southampton and other partner sites throughout the project, taking the group to Cambridge, London, Nottingham and Oxford. This strategy clearly helped maintain interest in meetings, spread the organisational load (and costs) and alternately reduced travel requirements for different members of the group.

A list of all project members is included in Appendix 3.


5  Year 3 Project Publications (chronological)

From May 1997:
Updated 15 December 1998
  1. Hitchcock, S, Carr, L, Harris, S, Hey, J M N, and Hall, W, 1997, Citation Linking: Improving Access to Online Journals. In Proceedings of the Second ACM International Conference on Digital Libraries, Philadelphia, USA, July (ACM: New York), pp. 115-122  http://journals.ecs.soton.ac.uk/acmdl97.htm
  2. Hitchcock, S, Carr, L, and Hall, W, 1997, Web Journals Publishing: a UK Perspective. Serials, 10(3), November, 285-299   http://journals.ecs.soton.ac.uk/uksg.htm
  3. Hitchcock, S, Kimberley, R, Carr, L, Harris, S, and, Hall, W, 1998, Webs of Research: Putting the User in Control. In Proceedings of IRISS'98: Internet Research and Information for Social Scientists, Bristol, UK, March  http://sosig.ac.uk/iriss/papers/paper42.htm
  4. Hitchcock, S, and Carr, L, 1998, Linking Quality Information Resources on the Web. A workshop presented at the UK Serials Group annual conference, Exeter, UK, March/April.
  5. Probets, S, Brailsford, D F, Carr, L, and Hall, W, 1998, Dynamic Link Inclusion in Online PDF Journals. In Proceedings of EP'98, the seventh International Conference on Electronic Publishing, Document Manipulation and Typography, St Malo, France, April  http://www.ep.cs.nott.ac.uk/~sgp/ep98.pdf
  6. Carr, L, Davis, H C, De Roure, D, and Hall, W, 1998, Application-Independent Link Processing. In Proceedings of the Seventh International World Wide Web Conference, Brisbane, Australia, April. Subsequently: Computer Networks and ISDN Systems, 30(1-7) http://www.elsevier.nl:80/cas/tree/store/comnet/free/www7/00/index.htm (frames version, Proceedings home page) or http://www.elsevier.nl:80/cas/tree/store/comnet/free/www7/1883/com1883.htm (no-frames, paper only) or http://journals.ecs.soton.ac.uk/~lac/www7/257veryshort.html
  7. Carr, L A, De Roure, D, Hall, W, and Hill, G, 1998, Implementing an Open Link Service for the World-Wide Web. World Wide Web, 1(2), 61-71

    8. http://www.staff.ecs.soton.ac.uk/~lac/imp.pdf
  8. Carr, L A, Hall, W, and Hitchcock, S, 1998, Link Services or Link Agents? In Proceedings of the Ninth ACM Conference on Hypertext, Pittsburgh, USA, June (ACM: New York)     http://www.staff.ecs.soton.ac.uk/~lac/LinksOrAgents.pdf
  9. Hitchcock, S, Quek, F, Carr, L, Hall, W, Witbrock, A, and Tarr, I, 1998, Towards Universal Linking for Electronic Journals. Serials Review, 24(1), Spring, 21-33 (an updated version of Linking Everything to Everything: Journal Publishing Myth or Reality? Presented at ICCC/IFIP conference on Electronic Publishing ‘97: New Models and Opportunities, Canterbury, UK, April 1997) http://journals.ecs.soton.ac.uk/IFIP-SerRev98.html
  10. Hitchcock, S, Carr, L, Hall, W, Harris, S, Probets, S, Evans, D, and Brailsford, D, 1998, Linking Electronic Journals: Lessons from the Open Journal Project. D-Lib Magazine, December  http://www.dlib.org/dlib/december98/12hitchcock.html
Or see the full list of project publications from 1995.


Overall project review (1995-98)

1  Project Review: Introduction

In contrast to the yearly reports, this section takes a broader perspective on the Open Journal project, its achievements - successes and failures - and discusses future prospects for the components, applications and link publishing strategies. In addressing these issues, the section is structured to enable those familiar and unfamiliar with the project alike to identify key points which might stimulate further investigation. This report can be read in conjunction with a separate, more detailed paper on lessons from the Open Journal project [10], or with the three annual reports to eLib, the last of which is included above.

The section is aimed at those publishers and Web information providers which might in the future want to build stronger capability for linking into their services. This might include developers and partners involved in the project because, although the project did not develop a commercial exit strategy, a number of initiatives have been set up between the partners to explore specific requirements with a view to building tools and applications that work in a commercial environment.

Links are an essential element of the Web, but link publishing - the ability to manage link information separately from content - remains an esoteric approach. Yet the first signs of commercial adopters outside the community that developed the techniques can now be seen. Setting aside the exact tools that might be used, and without specifying a timescale, it can be speculated that link publishing will become common practice in Web publishing applications.

It is not too late to investigate link publishing. Publishers involved in the project already have a head start. This is not an unreasonable return for participation, but it is stronger for two unplanned outcomes. First, instead of using a general-purpose tool set, publishers now seeking to build applications with the project developers are specifying custom software based on knowledge and experience of the project demonstrators and of their users' needs. Second, the project has no demonstrators that it can maintain publicly, so for those outside the project and who missed the earlier open demonstrations, it may be harder to understand what is happening.

For publishers the first sign that link publishing is emerging may be a new service introduced by a competitor. That threat became real just recently. As well as those publishers involved in this project, this report reveals another commercial journal publisher which is building its own link publishing technology and has announced a new development with possibly profound implications for other Web journal publishers.


Highlights of the project


Performance of the project against original proposal

Below are listed the objectives and deliverables outlined in the original project proposal.

Objectives

  1. To develop a framework of information retrieval technologies and electronic publishing practises to be used by information providers which will allow them to make their publications available not as isolated, one-off resources, but as co-operating assets within an information delivery environment such as a library at an institution of higher education.
  2. To develop agent technology for use within the framework to allow information to be negotiated automatically from a variety of on-line database resources.
  3. To use the framework to develop a set of ‘open journals’ based on existing publications.
  4. To run trials of the framework with publishers to determine its usability.
  5. To run trials of the resultant journals with users.
  6. To disseminate the framework and documentation to interested parties.
Against this measure the project was a success, with a significant proviso. A 'framework' was developed, used to demonstrate 'Open Journals' and run trials with publishers and users, and dissemination of the framework has begun too, if in a more limited form than envisaged.

What most limits the objectives is not the technology, but concerns the phrase 'co-operating assets'. The assets of journal publishers are still some way from cooperating, and the ability of software agents to 'negotiate' automatically with online databases must be preceded by negotiation between the content owners. Recognising this, the broader vision has been set aside for post-project exploitation in single publisher environments where negotiation between publishers is unnecessary, or in journal aggregation models where agreements on promoting assets are already in place.

Two possible developments may hasten wider moves towards cooperating assets as far as journals are concerned. First, shifts in the balance of ownership of content from publishers to authors and their institutions, stimulated by new publishing approaches such as e-prints, and use of these contents in multiple media, may prompt moves to widen ways in which new partnerships can cooperate. Second, there is evidence, perhaps indistinct at present, that the Web is evolving from a simple distribution network, a network of connected computers, towards distributed information and providing more support for 'interactive' information resources.
 

Deliverables

Developments in electronic publishing have rendered some of the proposed deliverables irrelevant. Almost universally, publishers now distribute their journals on the Web, formatted in Adobe's PDF, a process which began the year following the project launch and which has been tracked and reported by the project. What has been delivered is not a set of link databases for individual journals - linkbases were created for Open Journals collectively rather than the contributing journals - but a collection of software tools with which single journal linkbases can be built as required. A flexible, fast and effective bibliography agent has been demonstrated and can be applied as a standalone tool or in conjunction with the tool used to build linkbases. Producing documentation for these tools depends on improving the stability of the tools, on the range of applications, and on progress towards commercialisation.
 

2  An Open Journal framework: a concise summary

The Open Journal approach

In practical terms the Open Journal approach involved, respectively, linking, resources, formats and location: In this approach the link service has been integrated with two new components developed in the project:

Demonstrators and linking

Three demonstrator Open Journals were produced in the subject areas of cognitive science, biology and computer science. These areas and the core journal resources contained within the respective Open Journals were effectively defined by the interests of the developer teams and of the twelve or so publishers that supported the project (Appendix 2). It is more convenient, however, to consider the demonstrator Open Journals from the viewpoint of the linking features they highlighted (Table 3). The three main link types in this application can best be described as citation links, keyword links and PDF links.  
 Table 3. Open Journal demonstrators and linking features
Open Journal Citation linking Keyword linking PDF linking Release status
Cognitive Science Yes     Open release; closed end May 1998
Biology Yes Yes Yes Released to selected evaluators 
Computer Science  Yes Yes Yes Internal project release

 
The user view

User responses were gathered in a number of ways: meetings with individual users, selected evaluation groups and an open access demonstrator, mediated by electronic mail and Web forms. Typically, evaluators included specialists in the subject areas covered, and included researchers, teachers, librarians and publishers. Below are a selection of comments returned via a Web questionnaire in response to the open demonstration of citation linking in the Open Journal of Cognitive Science. The test results are presented in full in Appendix 1a.

"It's a great service!"

"It is a WONDERFUL idea. However..."

Important results from user testing


An Open Journal 'FAQ'

This section can be used to get a rapid overview of the project by those who may wish to investigate future applications.


3  Evaluation of the project

by Chris Bell, the project's external evaluator.

This section of the report presented a critical appraisal of the project's achievements. It described the aims and methodology of the evaluation, highlighted issues from which the project and others can learn, and made recommendations for future work.

The evaluation was included in the version submitted to eLib and circulated to project partners. We are unable to publish it here. If you are interested in this section of the project report and would like to see a copy of the evaluation, please contact the Project Manager, Les Carr.
 

4  Overall conclusions

What has been gained

Limitations

Could we have produced better Open Journals? Towards a journals 'gateway'

Yes, the demonstrator Open Journals could have been better. Mostly this answer is informed by results, as a development project should be, but other elements might have been foreseen. The Open Journals would have been better if we had: Although Open Journals are a way of aggregating large resources through linking, a complementary  approach which was not fully developed would have been to build the Open Journals as gateways, of the type that have thrived generally on the Web and elsewhere in eLib. One way of exploring this would have been to work with a subject-based information gateway project within eLib to combine technologies. There were attempts to do this but not enough time to develop the planned demonstrators. Of more relevance to publishers, however, would be a gateway focussed on journal resources, with link mediation provided by the Open Journal software.
 

Publishers: the way forward

The publishing background

Since 1995, when the project began, the number of e-journals available via the Web has jumped from around 100 measurably towards 3000, with perhaps more than 5000 worldwide according to some speculation (Table 4). Earlier estimates indicate that more than 80 per cent of e-journals are presented in PDF, which is the de facto standard for e-journals reproduced from print. It is against this background that the project's link application must be considered.
 
 
Table 4. The pace of change: summary of the growth of e-journals since 1995 
Oct. 1995 115 e-journals Hitchcock et al., Association of Research Libraries
Nov. 1997 c. 1300 'UK' e-journals Hitchcock et al., Serials 10(3), 285-299
 * Some predict we will reach 5000+ e-journals, worldwide, during 1998 
 
Recently, one publisher reportedly described the growth of PDF e-journals as the 'first frontier' facing journal publishers just a couple of years ago. The second frontier is the emergence of links, he went on to say.

It has not happened yet, but a path may be being mapped out in post-project applications with publishers. In turn, the success of these publisher applications may determine whether the link service software, or which elements of the software, are developed as commercial tools.
 

Post-project plans of participating publishers

Three publisher applications have begun, or will continue to be investigated, post-project. The dilemma for these applications, whether to continue with a static format such as PDF or to migrate to a more flexible format for Web publishing, is highlighted by two contrasting observations of the project developers: Clearly the latter has a bearing on immediate applications because it reflects existing publisher cultures and practices. Most of the continuing publisher projects involve working with PDF. These projects are supported by the original project developers but with a view to the publishers taking control and applying the tools themselves. All three approaches have one thing in common: they do not yet plan to use the full capability of the link service but instead will initially use a cut-down version of the tools: In addition there is the ongoing work with the JISC-funded Nature digitisation project to build link trails in a large PDF archive.

This all suggests that e-journals are not yet ready to become distributed, dynamically changing resources in a native Web format rather than PDF. This could change quickly, however. Publishers, now more familiar with working in multiple media, are attracted to the reusability features of the SGML format. Journal publishers in particular are formatting selected elements of published papers, notably header data, in SGML, and some have plans to generate HTML for the Web from an SGML original.

First examples of reformatting to support linking include conversion of reference sections to HTML, as demonstrated by the Institute of Physics e-journals service. This service, which uses PDF presentation, links citations from extracted HTML to a database of abstracts held by the publisher.

Alternatively, publishers may be tempted towards the Web based successor to SGML, XML, if it delivers more cost-effective production, particularly if e-journals can generate their own independent income streams to support this development. XML, and its linking component, potentially offers significantly more native capability for linking applications than does PDF, or even HTML, but it is not yet widely used. XML became a recommended standard early in 1998, and the linking components are not yet included in that standard. As a possible indicator to the nature of XML linking, research which pre-dated the project and which produced the link service software used in the project, was cited as an early influence on the standard.

There is belated recognition that e-journals must offer more than the printed equivalent. Citation linking will be the first example, and commercial agreements have been signed to do this in principle but not yet in practice. There are a number of possible effects. As more data is shared, how will it be managed, by whom and where? As shared data sources become larger, will static linking be adequate in a fast-changing, expanding data environment?
 

A platform for commercial tools

A version of the link service software is available from Multicosm Ltd, although it does not currently support journal applications as developed in the project. Negotiation continues with the company with a view to commercialising the link service for publishers, possibly with the additional components built in the project. This process will be informed by demand from publishers, particularly those experimenting with their own applications.
 

The final word: pushing back the boundaries

Although the application of project technology was not without its difficulties, and the demonstrators need to be followed up by real applications, a bigger barrier to adoption of an Open Journal approach remains cultural. The cultural shift required to embrace it is perhaps best highlighted by a description of the advantages of using a link service: Allowing others to exercise editorial control over already-published materials would appear to run counter to publishing ethos. This may be one reason why project publishers appear reluctant to unleash the full link service on their Web journals in the follow-up projects. Internationally there are political and legal moves to strengthen the exercise of rights and control over data published on computer networks. Yet nobody can know the impact that universal adoption of the Web, a uniquely user driven service, will have as a communications medium.

It is possible that anarchic users will stretch the limits of what is acceptable, but the motivations for change can be seen even among established publishers, who invariably have limited access to Web users. What if a publisher could extend its reach by allowing direct links to its works to appear in other services, library services for example? How could these links be maintained, updated and managed? Is it possible that respected publishers might want to do this, to interact with other services?

One is. The Institute of Physics' Stacks service - 'the ultimate linking service' - generates tables of contents (TOCs) with embedded hyperlinks, and is aimed at librarians, other publishers, aggregators, abstracting and indexing services and producers of information gateways. In contrast to the project's link service, Stacks delivers TOC and link data via email or file transfer to the local service provider.

Is this a more practical approach than the project has applied, more likely to appeal to publisher needs, or is it simply more limited and less flexible? Whichever, an important principle has been recognised by two developments independently: data, not just computers, are becoming perpetually more distributed on the Web. No data provider can survive alone. Data will be shared and interactive, and not just at the user level. The sooner this is more widely recognised, the more likely that established cultures can begin to change and efforts can be directed towards building an information environment in which new opportunities to serve users can flourish, rather than trying to constrain this environment by recreating and imposing other publishing models.

The legacy of the Open Journal project may eventually be commercial applications built by publishers and supported by commercial tools first tested in the project. Perhaps a broader legacy will be to have influenced developments leading towards distributed data, by motivating the user benefits at a time when the prevailing culture, especially among information providers, was difficult to reconcile with the emerging needs.


5  Open Journal Project Publications: the Complete List (by author)

Updated 15 December 1998

Carr, L A, De Roure, D, Hall, W, and Hill, G, 1998, Implementing an Open Link Service for the World-Wide Web. World Wide Web, 1(2), 61-71   http://www.staff.ecs.soton.ac.uk/~lac/imp.pdf

Carr, L A, Hall, W, and Hitchcock, S, 1998, Link Services or Link Agents? In Proceedings of the Ninth ACM Conference on Hypertext, Pittsburgh, USA, June (ACM: New York) pp.   http://www.staff.ecs.soton.ac.uk/~lac/LinksOrAgents.pdf

Carr, L, Davis, H, De Roure, D, Hall, W, and Hill, G, 1996, Open information services. Computer Networks and ISDN Systems (special issue, Proceedings of the Fifth International WWW Conference), Vol. 28, 1027-1036  http://www5conf.inria.fr/fich_html/papers/P12/Overview.html

Carr, L, Davis, H C, De Roure, D, and Hall, W, 1998, Application-Independent Link Processing. In Proceedings of the Seventh International World Wide Web Conference, Brisbane, Australia, April  subsequently Computer Networks and ISDN Systems, 30(1-7) http://www.elsevier.nl:80/cas/tree/store/comnet/free/www7/00/index.htm (frames version, Proceedings home page) or http://www.elsevier.nl:80/cas/tree/store/comnet/free/www7/1883/com1883.htm (non-frames, paper only) or http://journals.ecs.soton.ac.uk/~lac/www7/257veryshort.html

Carr, L, Davis, H, Hall, W, and Hey, J, 1996, Using the World Wide Web as an Electronic Library. Third international ELVIRA conference, Milton Keynes, May   http://www.staff.ecs.soton.ac.uk/~lac/elvira-full.html

Carr, L, De Roure, D, Hall, W, and Hill, G, 1995, The Distributed Link Service: a Tool for Publishers, Authors and Readers. World Wide Web Journal (special issue, Proceedings of the Fourth International WWW Conference) No. 1, Winter 1995/96  http://www.w3.org/pub/Conferences/WWW4/Papers/178/

Hitchcock, S, 1996, Web publishing: speed changes everything. IEEE Computer, Vol. 29, No. 8, August, 91-93

Hitchcock, S, and Carr, L, 1998, Linking Quality Information Resources on the Web. A workshop
presented at the UK Serials Group annual conference, Exeter, UK, March/April

Hitchcock, S, Carr, L, and Hall, W, 1996, A Survey of STM Online Journals 1990-95: the Calm Before the Storm. In Directory of Electronic Journals, Newsletters and Academic Discussion Lists, sixth edition, edited by D. Mogge, (Washington, D.C.: Association of Research Libraries), pp. 7-32, http://journals.ecs.soton.ac.uk/survey/survey.html

Hitchcock, S, Carr, L, and Hall, W, 1997, Web Journals Publishing: a UK Perspective. Serials, 10(3) November, 285-299   http://journals.ecs.soton.ac.uk/uksg.htm

Hitchcock, S, Carr, L, Hall, W, Harris, S, Probets, S, Evans, D, and Brailsford, D, 1998, Linking Electronic Journals: Lessons from the Open Journal Project. D-Lib Magazine, December  http://www.dlib.org/dlib/december98/12hitchcock.html

Hitchcock, S, Carr, L, Harris, S, Hey, J M N, and Hall, W, 1997, Citation Linking: Improving  Access to Online Journals. In Proceedings of the Second ACM International Conference on Digital Libraries, Philadelphia, USA, July (ACM: New York), pp. 115-122  http://journals.ecs.soton.ac.uk/acmdl97.htm

Hitchcock, S, Kimberley, R, Carr, L, Harris, S, and, Hall, W, 1998, Webs of Research: Putting  the User in Control. In Proceedings of IRISS'98: Internet Research and Information for Social Scientists, Bristol, UK, March  http://sosig.ac.uk/iriss/papers/paper42.htm

Hitchcock, S, Quek, F, Carr, L, Hall, W, Witbrock, A, and Tarr, I, 1998, Towards Universal Linking for Electronic Journals. Serials Review, 24(1), Spring, 21-33 (an updated version of Linking Everything to Everything: Journal Publishing Myth or Reality? Presented at ICCC/IFIP conference on Electronic Publishing ‘97: New Models and Opportunities, Canterbury,UK, April 1997)
http://journals.ecs.soton.ac.uk/IFIP-SerRev98.html

Hitchcock, S, Quek, F, Carr, L, Hall, W, Witbrock, A, and Tarr, I, 1997, Linking everything to everything: journal publishing myth or reality? ICCC/IFIP Conference on Electronic Publishing, Canterbury, UK, April  http://journals.ecs.soton.ac.uk/IFIP-ICCC97.html

Open Journal Project, 1997, Can links become the new publishing model for the Web? The Open Journal approach. A compilation of conference presentations  http://journals.ecs.soton.ac.uk/newpub/

Probets, S, Brailsford, D F, Carr, L, and Hall, W, 1998, Dynamic Link Inclusion in Online PDF Journals. In Proceedings of EP'98, the seventh International Conference on Electronic Publishing, Document Manipulation and Typography, St Malo, France, April
http://www.ep.cs.nott.ac.uk/~sgp/ep98.pdf

Probets, S, 1997, PDF, databases and the Internet. Acrobatics, 2(1), March, 3-7

For a classified and more up-to-date list of the complete publications see the project's Papers Web page.
 
 


 
 

Appendices

Appendix 1a. User results: Open Journal of Cognitive Science, version 2.1

Commentary

This open release followed two earlier releases to small groups of subject-specialist evaluators. Results from the first release were reported internally within the project and in the second annual report to eLib, but were not posted publicly. The second release was not reported, but resulted in modifications for version 2.1.

The principal changes in version 2.0 were an improved layout, other new presentational features, and speed of link serving. Version 2.1 saw the proxy setting added explicitly to the url, rather than requiring the browser settings to be changed.

Version 2.1 was notified to several large listservs and other mailing lists, and was publicly available from 28 April to 31 May 1998. Users were asked to complete a Web form, and 21 replies were received between 1-21 May. It is not possible to tell which mailings prompted most replies, nor to tell the backgrounds of the respondents.

The overall level of response was disappointing. Methods of communication have an impact. Email forms appear to be less effective than Web forms. There may not have been enough incentives for potential users. Otherwise, judgement on the relative response and its significance for the product being evaluated must be subjective.

Due to an error, one of the notices was mailed with the incorrect URL for the Web form, pointing those users instead to an earlier form. This form corresponds closely with the final version, but not exactly. Results shown in brackets below indicate replies submitted using the earlier form.

The results may not be statistically significant, but some observations can be made:

Ironically, the results understate the number of link errors in this release, which were discovered by simultaneous internal testing. There were two principal sources of these errors. First, the late change to the proxy-in-the-URL, which produced links that take users out of the Open Journal (effectively, in this situation users see the journal data in its original, unlinked form; it would not appear obviously wrong, but the Open Journal functionality is irrevocably lost). Second, inconsistencies in the citation style of the journal Behavioral and Brain Sciences which prevented the citation agent from correctly matching and linking the different resources. Both were tackled prior to release as far as deadlines would allow, but the time available was inadequate.

Yes, the form would have been revised had there been another release and would have been designed to offer more incentives, to better anticipate the users' interests and concerns, and to find out more about the users.
 
 

Responses to the questionnaire

1.  Set-up: Were you able to set-up and run the OJ successfully given the release email and  instructions on the Web?  Yes 12 (5)   No 0 (1)   Unanswered 3 (0)

2.  Presentation: This covers things like access to the journal contents and search facilities
a. Was it easy to navigate the OJ structure?  Yes 9 (5)   No 3 (1)   Unanswered 3 (0)
b. Can you suggest improvements or additions to make the OJ easier or more intuitive to use?

3.  Resources: all the resources listed on the contents page for the cognitive science OJ
a.  Is this collection a useful basis from which to explore the OJ concept?  Yes 8 (4)  No 1 (0) Unanswered 6 (2)
b.  'Wish list'. What online resources would you like to see linked to this Open Journal? (recommendations of freely accessible Web resources welcomed) 4.  Links
a.  Were you able to view and follow links served from the link server?  Yes 11 (5)   No 1 (0) Unanswered 3 (1)
b.  Were the added links useful?  Yes 8 (4)   No 2 (1)   Unanswered 5 (1)
c.  Would you want to change the way the links are presented?  Yes 2   No 8  Unanswered 5
How? d.  Did you encounter many broken links?  Yes 1 (1)   No 10 (4)   Unanswered 4 (1)
If yes, please specify URL and highlighted link copy.

5.  Reliability:
a.  Was the service available when you wanted to use it?  Yes 12 (5)   No 0 (0)   Unanswered 3 (1)
b.  Has the use of this service affected your non-Open Journal work on the Web in any way?
Yes 2   No 9   Unanswered 4
If yes, how?

6.  Your overall reaction:
a.  Did the Open Journal match your expectations?  Yes 9 (3)   No 2 (2)   Unanswered 4 (1)
b.  What did you most like and dislike about the OJ? c.  What changes would you make to the OJ to serve your needs? 7.  The way forward:
What would be your major proposal(s) for any future release of this Open Journal?

Acknowledgements

The project is grateful to all those who gave their views on this and earlier tests of other Open Journals and who contributed a vital element of the project's results.
 

Appendix 1b. User questionnaire


 
 
The Open Journal 
     Project 

Back to the Open Journal of Cognitive Science  

Electronic Libraries programme (eLib)

 We hope you have had a chance to view and explore this Open Journal. What struck you most about it? As you will have noticed, it isn't a 'journal' as such, but a way of aggregating, or linking, related resources in a way which ought to be useful for the reader. 

To encourage your views there is attached below a short series of questions. 

The primary purpose of this evaluation is to inform further development of this and other Open Journals. Your responses to the evaluation will be formally reported to the project's developers, partners and publishers. A summary report will also be forwarded to the directors of the Electronic Libraries (eLib) programme, the body funding the project in the UK, to inform their drive for integration between eLib projects where possible and for sharing of information with other projects. A copy of the final report to eLib will also be made available on the Web during the summer. If you wish to be alerted when this report is available, please email Steve Hitchcock, heading the mail 'Send final report alert'. 
 

This is still a system under development. If you have had problems, our apologies, but please do report them since this aspect of the evaluation is as important as any other. 


EVALUATION QUESTIONS 

This framework covers issues such as set-up, presentation and the interface, journals and other resources, links, reliability and your reactions and thoughts on future work. 


1.  Set-up 
Were you able to set-up and run the OJ successfully given the release email and instructions on the Web? yes no 

2.  Presentation 
This covers things like access to the journal contents and search facilities 
a. Was it easy to navigate the OJ structure? yes no 
b. Can you suggest improvements or additions to make the OJ easier or more intuitive to use? 

3.  Resources: all the resources listed on the contents page for the cognitive science OJ 
a.  Is this collection a useful basis from which to explore the OJ concept? yes no 
b.  'Wish list'. What online resources would you like to see linked to this Open Journal? (recommendations of freely accessible Web resources welcomed) 

4.  Links 
a.  Were you able to view and follow links served from the link server? yes no 
b.  Were the added links useful? yes no 
c.  Would you want to change the way the links are presented? yes no 
     How?  
d.  Did you encounter many broken links? yes no 
If yes, please specify URL and highlighted link copy 

5.  Reliability 
a.  Was the service available when you wanted to use it? yes no 
b.  Has the use of this service affected your non-Open Journal work on the Web in any way? yes no 
If yes, how? 

6.  Your overall reaction 
a.  Did the Open Journal match your expectations? yes no 
b.  What did you most like and dislike about the OJ? 

c.  What changes would you make to the OJ to serve your needs? 

7.  The way forward 
What would be your major proposal(s) for any future release of this Open Journal? 

Your Name:    Your Email:  
 

"Many thanks for your continuing help and support." 
 
 

Appendix 2. Publishers involved in the project

This is the complete list of publishers involved during the project: For publisher links see the project's Partners Web page
 

Appendix 3. Project people

This is the complete list of people who were at some stage involved in the project: For contact details see the project's Personnel Web page
 

Contacts

About the project; about this report
Steve Hitchcock  sh94r@ecs.soton.ac.uk
About the link service software
Les Carr  lac@ecs.soton.ac.uk
About PDF linking
Steve Probets  sgp@Cs.Nott.AC.UK
This page: http://journals.ecs.soton.ac.uk/yr3/3rd-year-open.htm
[Top of page]  [First year report to eLib]  [Second year report to eLib]  [About ... the Open Journal Project]  [Open Journal Project]