[AV Media Matters] Setting the Agenda first AMIA-DPC

[e.ha.zwaneveld@nfb.ca]

Dear Colleagues,

First, thank you for joining this specialized electronic AMIA Preservation
Committee Work Group. Second, thank you Jim Wheeler for chairing it and Jim
Lindner for enabling it to be done electronically.

I am mailing this document also to AMIA-L, AV Media Matters-L, SMPTE-V16.07
and EBU/IFTA FTA WP-1, to signal the start of our Adiscussions. We will
provide updates from time to time to those other listservs, but discussions
will take place under AMIA-DPC. I would like to invite any non-AMIA members
to channel their feedback to me, so we will be able to enlarge our dialogue
and add depth to our discussions with your inputs. Those of you who wish to
enjoy the full discussion are encouraged to join AMIA. You can find its
Homepage: www.amianet.com where all the necessary information can be found
to join. Then let me know when you are ready please!

We would like to agree on a global approach to meet our two objectives: (1)
to draft an AMIA Digital Archiving Policy, to help guide our members through
the labyrinth of video archiving and (2) to provide feedback on the
comprehensive search for archiving solutions by the European Broadcasting
Union (EBU), the largest video users group in the world, representing 50
member countries and another 30 countries who are associate members.
Broadcasters and video post-production houses are the largest sources of
archival material to end up in the hands of archivists, or with a more
modern word, Digital Audio-Visual Asset Managers. So it is of great
importance that your voices be heard and noted in their search for
solutions, because if they fail, you will have to live with the
consequences. At the end of this e-mail I would like to propose a list of
headings for our Digital Preservation Policy and invite additions as
appropriate.

But let us respond to Jim Wheeler, who on January 28, gave us some
challenges to put our teeth into, which I will do gladly. Jim listed several
candidate archiving formats, including film (expensive), Betacam SP (analog
and is near the end of its life), D5 and D6 (high quality formats but
expensive) and DVD (uses heavy lossey compression).

(a) Having said that, I would suggest that this list should also consider DV
and MPEG-2 422 50, which are clonable, their compression is conservative if
it does not exceed 3.3:1 and can be considered lossless if the source going
into those formats is from compatible families. They also have the advantage
of providing faster than real-time migration at some future date, which
saves time and money to archivists.

(b) I contend however that if film were able to be used for storage of
video,  would be too expensive, it actually is less expensive than
videotape! Consider the cost of migrating video collections every 10-15
years, just because the recording system is obsolescent! That is not cheap,
and adds up when you have to do that 6-10 times in a century, hoping against
all hope that the 'digital' wonders, typically written in proprietary
dialects, will talk to and be understood by each future link in the chain
and come out of it all with our content integrity intact. Expensive and an
unsustainable nightmare if you ask me, certainly not cheap or affordable.

(c) So it is important that we consider all angles before we consider that
the solution is the building of a simple archival machine.

(d) At the moment we appear to have only magnetic tape as the most suitable
storage medium, but its binders are not able to provide Extended-term Life
Expectancy, while the technologies required to play it back are by design
short-lived and users typically do not store the tape under the best
conditions.

(e) Magneto-optical media are considered the most archival media on the
market today, although they are in disc form and require even more
compression than tape. However, a Kodak Digital Optical Write Once Read Many
times (WORM) tape is on the horizon inside the next 1-1.5 year when it will
be announced. It is designed to last more than 100 years. (Although I would
say that it will require rigorous qualification testing by at least two user
laboratories to verify, publish and its manufacturer (s) to certify that it
will last 100 years or more. It will be designed to store digital data, such
as for digital motion picture masters, so it is likely to also be suitable
for digital data representing images and sound. The beauty of it is that
this company has a track record with universal standards for 35mm and 16mm
film that has been supported for 100+ years already.

(f) So to the question: "Do we assume that magnetic tape is the only digital
media with the potential of being archival...", my answer is that magnetic
tape has not been designed to be archival if you mean to say as having
extended-term Life Expectancy (LE). It remains a temporary storage solution
because it must be inexpensive. I do not believe that a binder can be made
that will have a Life Expectancy of 100+ years. Manufacturers are unanimous
that any product on the market today should not be expected to live beyond
20 years. And no binder with longer Life Expectancy has been offered, tested
and certified to do that, so we cannot assume this. Until it is here and has
been subjected to thorough tests, published by at least two credible user
laboratories and certified to be archival by the manufacturers, I will
consider that assumption as no option.

7. Should the specific issues of compression be addressed by our policy, I
feel it should.

8. Should ultra-thin tapes be addressed? I believe they should.

9. The statement (7) that: "The TV Broadcast Industry believes compressed
formats are archival. Should we develop a new term in place of the misused
word archival?  Our Webster Dictionary states that an Archive is: 1. (a) a
place where public records, documents, etc. are kept; (b) a place where
material having documentary interest, as private papers, institutional
records, memorabilia, photographs, etc. is kept. 2. the records, material,
etc. kept in either of these places. Something is archival: of, in, or
containing archives.

This definition is fine, so everything kept in archives is archival. So TV
Broadcasters are perfectly welcome to use the term. The problem is that the
term has been misapplied to indicate that something that is archival 'can
stand the test and ravages of time'. The other problem is that it has
typically been associated with the 'physical artifact'. Until digital
technology was developed, a 'replica' was not considered to be an acceptable
replacement for the real thing. But if digital technology is applied
correctly to the content of the record, it can provide a lossless clone of
the original source. Until its entry into the world, audio visual records on
magnetic tape media used 'lossy and sloppy analog' recording technologies.
And because we consider full content integrity as the present objective, any
comprehensive solution that guarantees Extended-Term 'content integrity
preservation' Life Expectancy is acceptable as it can stand the test of
time. That emphasises the thing that really matters to us, the 'audio-visual
content' of the document that has the potential of an Extended-Term Life
Expectancy (LE).

To support the above argument I note that our old mindset has perhaps been
infected with an outdated and non-applicable concept. We should develop a
new definition that matches our 'digital clone' ideal, so we won't waste
scarce money on lossy temporary methods.

I quote the 1990 ANSI IT9 Committee which has jurisdiction over ANSI
standards pertaining to physical properties and permanence of photographic
materials and other imaging media. They then voted to abolish the
long-standing "archival" designation in these ANSI standards. For films,
they replaced the "archival" designation with Life Expectancy ratings (LE
ratings), which are given as "years of useful life" under specified
processing and storage conditions. In recent years the same terminology
change has taken place in magnetic media standards. The ANSI/NAPM
IT9.23-1996 does not use or define "archival" any more . But it does define
Life Expectancy (LE): "The length of time that information is predicted to
be retrievable in a system.

The same document defined Extended-Term Storage Conditions:" Storage
conditions suitable for the preservation of information recorded on the
majority of magnetic tapes for 50 years.").

A good source manual is: Henry Wilhelm, The Permanence and Care of Color
Photographs: Traditional and Digital Color prints, Color negatives, Slides,
and Motion Pictures", Preservation Publishing Company, Grinnell, Iowa, ISBN:
0-911515-00-3, 1993.

On page 92 of his book, he quotes ANSI in 1990:
"Archival medium: A recording material can be expected to retain information
forever so that it can be retrieved without significant loss when properly
stored. However, there is no such material and it is not a term to be used
in American National Standard material or system specifications."

Life Expectancy (LE): "The length of time that information is predicted to
be retrievable in a system under extended-term storage conditions."

So here are a few proposals to get started on our Digital Preservation
Policy:

PROPOSITION: 1
Life Expectancy (LE): "The length of time that information is predicted to
be retrievable in a system under extended-term storage conditions."

PROPOSITION: 2
Extended-Term Content Preservation Conditions: "Digital content preservation
conditions suitable for recorded information having permanent value with
unlimited Life Expectancy".

PROPOSITION: 3
Medium-Term Content Preservation Conditions: "Digital content preservation
conditions suitable for recorded information with a Life Expectancy of a
minimum of ten years."

So we need to redefine the Extended-Term Life Expectancy to reflect our
Digital Preservation Policy need.

PROPOSITION 4:
Extended-term Life Expectancy: "Standardized recording system conditions
that permit audio-visual content to be transparently and losslessly
interchanged regardless of the production/post-production or the storage
system, the medium type, the encoding format or the transport solution that
has been applied to it or the time that has passed since the source content
was created".

Finally, I would like to propose the following potential topics for
discussion, some of which would become headings for our Digital Preservation
Policy:

(1) Types of archives-dynamic, dormant, virtual; (2) What will future
archives be like? (3) Automation is imperative for future asset maintenance;
(4) Why we have a legacy of old tape formats; (5) Storage formats; (6) The
cassette and tape; (7) The video compression schemes; (8) Audio criteria;
(9) Automation functionalities; (10) Format extension; (11) Indispensable
functions for future archival automation; (12) Making archives
automation-ready; (13) Transferring legacy video tapes; (14) Quality; (15)
Systems; (16) Criteria old archival material and its related playback
equipment are to meet; (17) Content Preservation Sustainability; (20)
Standards that must be developed to protect Extended-Term Life Expectancy of
Content.

Those of you who are interested in my position paper, presented in behalf of
AMIA at the recent Joint Technical Symposium (JTS) in Paris, feel welcome to
request it by sending me an e-mail.

I look forward to an enlightening discussion and great propositions to
consider.

Ed H. Zwaneveld
Technological Research and Development
National Film Board of Canada and
Chair AMIA Preservation Committee