Repository storage: the impact of data growth and diversity

March 2009 - Once content has been ingested into the repository, it has to be stored and available for access and presentation. This section considers the function of storage in digital preservation, focussing on work done by the Preserv project.

Computer storage should ideally be cheap, expandable and reliable. A virtuous coincidence of circumstance has allowed us to benefit from the first two factors for some time. As storage hardware costs fall, however, the volume of digital data continues to grow rapidly. As this happens it has become clear that the cost of managing data is more important than hardware costs. In addition, as volumes grow costs can rise to achieve specified levels of storage system reliability (that is, systems with measured probability of digital bit-flips) for storage of particular types of content such as science data. Faced with an increasing range of storage options - large-scale storage services, 'open' storage (cf. open source software), cloud storage - our choices have to take account of these factors as well as to control costs.


Computer applications that manage storage on a large scale thus need flexibility to choose, manage and change their storage options. It may not feel like it now, as some struggle to attract content, but repositories are such a case and need to prepare and plan. This section reviews the emerging technologies that Preserv has investigated and developed to widen the storage management choices available to repositories, and includes storage controllers, network 'cloud' storage and 'open' storage

To understand why digital content might be harder to manage than physical artifacts, we have to consider all types of data. Problems handling dynamically-changing data are revealed through a classic 1957 sketch from The Goon Show.
Tutorial presentation Preservation and storage management for Institutional Repositories (slides), Repositories Support Project Summer School, June 2008
Explores the relation between different types of data - images, audio, as well as institutional data types such as Web sites and repositories - and the needs of storage and preservation Summary. There is an associated group worksheet.