SNIA DMF 100 Year Archive Requirements Survey Highlights

In September 2006, the SNIA Data Management Forum (DMF) 100 Year Archive Task Force determined it needed a clear statement of business requirements to frame and bound potential technology solutions to the growing digital crisis of long-term information retention in the data center. To meet this need, the group conducted the 100 Year Archive Business and IT Requirements Survey, inviting a broad range of information owning and administrating professionals worldwide to participate and provide guidance. The full report is now available, and can be downloaded for free at www.snia-dmf.org/100year.

The Digital Crisis

Large organizations faced with retaining and preserving huge amounts of digital information for very long periods of time are at the front edge of a troubling crisis.

These data centers can be characterized as having environments with petabytes of distributed information, high data growth rates, many facilities and many departments with uncoordinated responsibilities and requirements, and lack of business-level budget, interest, and focus on archives. All these operating challenges are now compounded by a high risk of failure and fines from legal discovery, compliance requirements or security threats. Add to this, the risk of losing information that may be of great value to the organization and the picture looks pretty daunting.

Top Four Ways of Losing Digital Information

• Cannot read it
• Cannot interpret it correctly
• Cannot validate its authenticity
• Cannot find it

In addition, the digital crisis is exacerbated by time. In 10 years, 50 years, 200 years, which applications will still be around? What computer and storage system will be able to read old information? The problems are huge and the dilemma is that standards and best practices do exist today documenting the practices of preserving digital information. Yet, none of them address the core problems caused by inadequacies and inefficiencies in the supportingstorage infrastructure.

It is the contention of the SNIA DMF 100 Year Archive Task Force that new technological approaches are required that meet the legal, business, cost and scalability requirements of the digital age for long-term digital information retention.

Key Survey Findings

The respondents were very insightful and provided opinionated and experienced advice. Here are some of the important highlights, with page numbers provided to reference the full report for more information:

• The survey establishes clear validation that long-term retention needs are real and that many organizations have long-term requirements.
  -80% of respondents declared they have information they must keep over 50 years, and 68% of respondents said they must keep it over 100 years. (See page 33)
• Long-term generally means greater than 10 to 15 years - the period beyond which multiple migrations take place and information is at risk. (See page 23)
• Database information (structured data) was considered to be most at risk of loss. (See page 34)
• Over 40% of respondents are keeping e-Mail records over 10 years. E-Mail is not just a short-term problem. (See page 35)
• Physical migration is a big problem. Only 30% declared they were doing it correctly at 3-5 year intervals. The rest of the sample group is placing their digital information at risk. (See page 38)
• 60% of respondents say they are highly dissatisfied' that they will be able to read their retained information in 50 years. (See page 46)
• Help is needed - current practices are too manual, too prone to error, too costly and lack adequate coordination across the organization. (See page 40-46)
• Collaboration and classification were recognized as very important practices to get the organization working together setting requirements for the management of their information. This recognition validates the messages of the DMF in its market educational efforts for Information Lifecycle Management (ILM)-based practices. (See page 53)
  - Reinforcing this point, only 35% of respondents agreed with the statement that their IT and RIM departments coordinate requirements for retention and preservation of the information they protect. (See page 44)

The Survey

• Online, Quantitative Survey
• 276 Organizations
• IT, RIM, Archivists, Legal, Security, and Business Respondents
• Worldwide Participation

Follow On Work By The SNIA-DMF 100 Year Archive Task Force

The SNIA DMF is working to improve the art and science of digital archives. We are working on:

• A storage focused, long-term archive reference model that includes automating data migration in self-healing environments
• An archive storage architecture that will leverage the ISO standard OAIS on logical archive formats to ensure long-term accessibility and readability
• Using XAM (the eXtensible Access Method interface under development at SNIA) as enabling technology to compel application developers to create solutions that are developed archive-ready. Solutions that will allow archive data to migrate seamlessly between competing vendor storage infrastructures, without application knowledge, while maintaining long-term physical and logical readability.

Get Involved

If you want to contribute to this project, participate in the 100 Year Archive Task Force, which is part of the SNIA Data Management Forum. You can help guide this work and elevate its effectiveness. Solving the long-term digital information retention and preservation challenge is very important, and the Task Force needs experienced participants from many disciplines to address the complexity of these problems. You can learn more, including how to get involved, at www.snia-dmf.org/100year.

¹OAIS: Open Archival Information System, ISO Standard 14721:2002

²XAM: eXtensible Access Method - a new standard in development by SNIA