Smart storage

March 2009 - The Big Picture for Preserv combines our approaches to repository management and data, storage and preservation services. We call this approach 'smart storage':


So far this approach has been developed to supplement one part of the preservation format services, format identification, as shown in this figure. Additional tools - a calendar-based scheduler, a harvester to retrieve the latest stored content, messaging services to record the history of events and the results, all designed to work with, or 'wrap', the core format identification tool DROID - combine to manage and process the data and results.

Smart storage has been elaborated in two presentations and in a paper, aimed at the technical preservation community, and showing what libraries or repositories can do.

Presentation Smart Storage and Preservation: How Digital Repositories Can Participate (slides), DLF Fall Forum 2008, Providence, RI, November 2008
Aimed at repository and library managers, the presentation illustrates a flexible and practical approach for technical preservation activities, such as handling format issues, based on modular services. By dealing separately with items in the preservation workflow - format identification, planning and risk assessment, and action (including migration) - a 'smart storage' approach based on a series of autonomous Web-based services, combining massive storage with the intelligence provided through the respective services, is being developed. The presentation identifies tools and services that can contribute towards smart storage, and shows how repository managers can specify and select components based on non-technical, policy-led considerations.

Paper and presentation Towards smart storage for repository preservation services (paper and slides), iPRES 2008: The Fifth International Conference on Preservation of Digital Objects, London, September 2008
Examines at a conceptual and practical level how preservation intelligence can be built into software-based digital preservation tools and services on the Web and across the network ‘cloud’ to create ‘smart’ storage for long-term, continuous data monitoring and management. Some early examples are presented, focussing on storage management and format risk assessment.

Download software Preserv2 EPrints preservation toolkit
Allows a repository running EPrints v3.2 software (due 2009) to produce its own profile of the files in the repository. This includes file type as well as related risks to these files. Includes a DROID wrapper, caching database (as an EPrints dataset) and results page (as EPrints Admin Screen). For more description on how this toolkit works see this Preserv wiki page on EPrints and preservation.

Calendar server

The illustrated Preserv approach to smart storage uses the iCal calendar standard in an implementation called Darwin Calendar server. This controls the timing of plannned preservation events, such as file format management, and records the results. The scheduler can also be used to manage preservation actions such as scanning for viruses or malware, and can handle recurring events. See this blog by Preserv contributor Ben O'Steen: "Most actions that result from a preservation policy choice can be pre-assigned to an item when it undergoes a change of state (creation, modification, deletion, extension)".

The possible inclusion of a scheduler in EPrints v3.2 is being investigated.