The Open Citation Project - Reference Linking for Open Archives

V 1.0 demo: evaluation by the OpCit steering group: results

Totals

Summary of author results and main findings.
 

Evaluation

How effective is this as a demonstration of reference linking?
Very xxx 3
Moderately xxxx 4
Hardly 0
Not 0

• Moderately. I don't like getting linked to PDF documents; I have to invest a minute of download time to decide if I'm interested. I'd also like to go to the journal, not to a preprint, when I have the option.
• Moderately. The problem is, when I tried one of the papers (with SFX on) and clicked on a red box, it said "A web browser has not been specified". Going through the procedure didn't really help. Also, there were no links in the text itself (orange boxes?) only red boxes at the end, which made this a less effective demonstration.
• I tend to say "very" when my comments re the SFX explanation are considered. Also, I feel that ScreenCams could provide some additional value. But, from my own experience with that, I feel that it would be great to have something that would stream.

What features need to be improved?

• Better search
• I don't expect to be able to use a search facility when I'm linking directly from a citation to an article. This is a different scale of things - e.g. it's like trying to be an Ingenta or ISI.
• Better search would be nice
• I didn't see a search
• Do you mean searching for links within a document? I'm confused -- I didn't think there was any searching going on for users.
• It is somewhat awkward to break an arXiv id into year, month, and article number, it should be a single text entry box. If you stick with the current scheme, the article number box should add leading zeros as needed. If you go with a single box, the Los Alamos way of allowing current year or current month to be inferred should be supported (i.e, entering a '1' should correspond (today) to 0007001, entering a 6001 should be 0006001, etc.

• Link reliability
• So far, in the demo, it's really good. You know from looking at a link if it will work or not. However, in future versions, I'd rather only see the links that actually work, i.e., before making the link, make sure that you can link to the target doc.
• Journal links don't seem to work well
• Didn't test this too extensively. However, picking a paper at random (OK, it is one of mine), hep-th/9201076 all links have incorrect data (just a year for preprints, no journal name for journal articles) - so links don't work. Colors also seemed incorrect - for the two preprints cited, they should be orange and not red since the arXiv ID wasn't there.
• Link reliability: quite an issue, isn't it? Actually a very interesting research matter on its own.

• Link presentation
• presentation is great. You might want to think about making the red boxes into blue boxes, since blue is the default link color for most browsers, but that's nitpicking.
• a link button with a graphic would be more transparent to use
• The orange and red boxes are acceptable, although my guess is that highlight the link in red or orange rather than putting a box around it would look better.
• I would really like to see the links available either within the PDF OR separately as part of a wrapper page. Following link trails is inconvenient if you have to download a full PDF every time rather than going from wrapper page to wrapper page. Being able to extract links from a PDF into some kind of a tagged XML format (including reference type and tagging the components of the citation) would a very valuable tool. For the demo, having coded boxes for reasons things don't link is fine. Presumably in a live version, you would just not display anything for this references.
• I am colour-blind -- like about 50% of the men on this planet -- and this colour indication is really problematic for me. One could also think of indicating these things with figures. Have you seen the way the CNRI people show handle links on their own pages? Quite inspiring.

• More links
• in our HTML articles we make internal article links. Links from a figure mention ("See Fig. 1") to the figure, from a footnote mention to the footnote, from a reference mention to the reference. You could try to make those intra-document links, but let me tell you they're a lot harder than just reference links. I think that reference links are probably the number one bang-for-the-buck link you can put in scientific articles.
• More links in the text would have been nice
• link from main text to refs would be good
• Links to SPIRES and ADS would be great.
• addition of alternative resolvers i.e. other servers that provide service links (see OpenURL) addresses the "more links" issue.

• User interface
• fine for me, but you should probably do some usability testing, if you haven't already.
• I'm not convinced that I found the SFX on/off feature very useful - it just seemed to get in the way, but this probably says more about my lack of appreciation of the "appropriate copy" problem.
• I had problems with the user interface.
• The SFX-type page should display detail of the object it is trying to find and be a little more smart about what buttons are offered.
• Should be a way to choose which arXiv mirror you go to. Links shouldn't go from PDF to PDF. Much better to link to a wrapper page (link LANL abstract page) that gives users more information (abstract, PDF size, etc.) before final download decision. SFX page should give more information about the two papers being linked - at least display the arXiv id's (but full reference metadata would be even better). SFX base URL should be settable to point to other resolvers. OpenURL spec should probably be adopted. SFX page could use a lot of prettification. Adding a discovered arXiv id to reference might be a nice feature.
• I feel that your own SFX menu-screen could use some cosmetics. On http://arabica.ecs.soton.ac.uk/demo.html , I suggest to put the header for the explanation of what SFX-style is also on top of the page, not in the table. Plus, it would help to say -- in the explanation on SFX-style -- that "on" means the user will get a list of extended services (including full-text links), while "off" means that only links to full-text are provided. In the SFX explanation, the sentence re "SFX in principle, not SFX as such" is very confusing. Why not say that the provision of extended services via an intermediate menu-screen fits into the SFX framework, but that in the demo you don't actually use an ExLibris SFX server.

• Other (specify)
• Your highlighted demo PDF files contain bitmapped fonts rather than Type 1 fonts. This should be fixed. PDFs of my paper I downloaded were Type 1 though.
• Many of the PDFs are very poor quality (produced from scanned images rather than Adobe Capture or directly from the origination program, I guess)

• All these should be handed off via virtual links such as LinkBaton.
• I don't think you have to provide all of this yourself. Just open up your environment via OpenURL, so that others can demonstrate their creativity in creating linking services.
• Links to online journals are our #1 concern. In terms of the demo, I don't know what databases you might link to, but in our biomedical articles, we have been asked to link genetic sequences to GenBank and some article references to PubMed. These are both nice features, and will be required at some point, but as I said above, the reference linking is most important for now. The main thing that we would need added to the software before we could use it would be much better linking to online journals. That includes:
• A. recognizing the references to each journal, in spite of authors who seem to be trying to make it difficult by using weird abbreviations for things.
• B. algorithms for creating article links into each online journal. Our journals all work the same, but many other journals don't. Since you can't really take the time to test each link before you place it, you'll need a list of online journals that you can link to and the method to turn volume, issue, and page into a URL for each one.
• C. possibly link status icons, based on journal info. For example: a free journal would have a little icon next to links to it, notifying people that they don't need a subscription to view the link. Subscriber-access only journals might have a different icon (or no icon, since this is pretty much the default :( ), and journals with pay-per-view might want to advertise that in links.
• We've done some work on A., B., and C., at least for our own journals

• We would like to run all of our PDFs through the software to generate reference links to all of our other journals. Secondly, we'd love to link to other publishers' journals and to PubMed. Additionally, we are looking into Manuscript Submission and Review systems right now, where an author would submit a PDF of his paper and the journal staff would review it online. If we could link the references in that (totally randomly formatted) paper, it will save reviewers tons of time in checking those references.
• Its most direct relevance is a) to my work on DC-CITE, and b) to CrossRef.
• If you can provide a simple PDF->PDF+OpCit converter that puts in SFX-type links then I see no reason why we couldn't incorporate this in arXiv at an early stage (we give two PDF options: PDF and PDF+OpCit links). If an SFX-type server is used then there would presumably be no need to for us to store any resource databases. We are currently testing OpenURL linking from our abstract pages (http://sfx1.exlibris-usa.com/openurl/openurl.html). It would be good to be able to specify a BASE-URL (taken from cookie) which gives the user's chosen server as input to such a converter. However, I imagine this creates a whole bunch of issues about whether you want to use the type of ids/parameters specified by OpenURL. Our PDF usually has links already: either within the document, to other arXiv documents, or to URLs. I would like to see the internal links preserved as mentioned below.
• The free-form reference parsing and dynamic adding of references to PDF would be great for handling manuscripts within the editorial process (pre-production). Even in production it would be helpful to add links to our PDFs (though there is an argument against this - adding links is a dynamic process and is viewed as an added-value service - making them part of the PDF rather than a wrapper page means 1) a user looking at a saved PDF may not be getting the most up-to-date links and 2) they are able to retain the added-value service after their subscription lapses. SFX style linking addresses both of these problems though (a user would have to go back to the publisher gateway and this could be subscriber controlled). This is not necessarily the dominant view around here - having links in the PDF is certainly convenient, but I thought I would mention it.

• we might be able to set up a service that would link PDFs for other people.
• I would very much like to see CoRR hooked into this.
• Support for either DOI linking or pragmatic citation linking (APS link manager) for instance would be a valuable addition. APS could work to supply accurate metadata for improved matching of Phys. Rev. citations
• wouldn't it be nice to let users choose an SFX resolver. I can at least think of 3: your system, our SFX server in Ghent and a demo SFX server from Ex Libris. This approach would clearly illustrate the place of the SFX framework in reference linking. And it would somehow make the mentioning of Ghent and SFX as partners in OpCit more real. Implementing this possibility only requires the implementation of the CookiePusher and the OpenURL (see http://www.sfxit.com/OpenURL). In most environments that we have dealt with so far, the work involved is marginal. Initially -- to make things simple -- it could even start by only providing OpenURLs for arXiv identifiers. As you can see from the specs, the OpenURL is extremely simple in this case.

• For most tex papers our current system uses hypertex to provide links within papers and to other archive articles. Clearly your approach is more general and will deal with the increasing number of non-tex submissions. I'd like to see your system produce the intra-paper links from references to the references section (though I imagine some people would prefer the reference to go directly to the referred-to paper - actually less useful in my opinion since in must cases one just wants to know which paper/worker is being referred to). I see no reason why you should not create links to journals and archives for references that cite both. Obviously, linking between archives is relatively straightforward (though I notice that references split by a line-break are not found, eg [132] in hep-th/0001001). I think the real test is identifying references to journals and perhaps even books.
• did you know arXiv now IS OpenURL-aware? The Ghent SFX-server already provides quite some services for incoming OpenURL's carrying arXiv identifiers. We fetch metadata back from arXiv using the OAi Dienst Subset.
• I am quite interested to understand more about the technique you use to actually insert links in the references: are the links being re-written dynamically at every download, or do you have a link being inserted once (in an batch process) that points at a front-end that connects to something like a handle system for the real resolution of the link (the DOI-style approach)? The more I work on these matters, the more I feel the latter is a very interesting path. With this respect you may be interested in our recent DOI-related experiment documented at http://sfxserv.rug.ac.be:8888/public/xref/ , and to be published as soon as I have time.