Hey Tom,
I believe every archive has been bit by some sort of migration issue, and basically, you can trace it back to decision making that wasn't either well planned, understood, or funded.
I've heard that before my time at EMI, a few large scale digital initiatives were started and abandoned. As I said, it was before my time, so I don't actually have the facts, just the passed on oral traditions.
But migration is life, as no media lasts indefinitely. We are however in a much better position digitally that we were 10 years ago. Converters and file formats rival the Sonics of analog. Media/format solutions such as LTO are far more accepted and open source than Exabyte, and there are other industries which are struggling with the same issues as we, therefore there more people working towards a solution that could be used to benefit all. But most importantly, I feel that after learning from past mistakes, we can all see the importance of planning a project through the initial phase of the transfer, and building in metadata and future migration paths from the impetus.
Those Exabytes may have been saved if there was follow up money allotted up front. The problem was that in most cases all the money for migration was spend on the transferring. There are many fields (medical and financial come to mind) that have been migrating digital data for 20 years. They've used Exabytes, and they got off them before their data went bad.
That's why I advocate NOT transferring until you've really thought it through, and secured the funding you'll need to see it through to the next phase. Anything less is a wasted effort.
Don Andes Director of Archives EMI Music
-----Original Message----- From: Association for Recorded Sound Discussion List [mailto:ARSCLIST@xxxxxxx] On Behalf Of Tom Fine Sent: Wednesday, May 16, 2007 4:11 PM To: ARSCLIST@xxxxxxxxxxxxxxxx Subject: Re: [ARSCLIST] Mass Digitization
Hi Don:
Somewhat related to your posting, I have a question. You probably have hands-on experience with the topic.
In the mid-90's I remember at one large mega-glomerate's remastering
facility, exabyte (sp?) became all the rage. The argument was made to
"clone" all the 1630 masters and then throw them out. But, a couple of
years in, the exabyte tapes(?) started malfunctioning randomly. Luckily,
the wiser producers insisted on keeping their 1630 tapes in the library,
so no harm no foul in those cases. But some pretty darn good remasters done in the late 80s onto 1630 and
then "cloned" to exabyte and the 3/4" 1630 tape reused or discarded were
lost and had to be done over. Some of this was all good because the
master tapes were still in good shape and the new remasters were better
anyway. But in some cases, the tapes were lost or had been in bad shape
10 years earlier and were unplayable by the late 90's.
Did you guys have any situations like that?
I tell this tale because exabyte was apparently the migratory flavor of the moment at one time and we should keep stories like this in mind as we migrate digital archives over time. The lesson I take away is that the latest might not be so greatest so best to run parallel formats (old and latest) at least until some knowledge can be gained of the weaknesses and pitfalls of the new format.
-- Tom Fine
----- Original Message ----- From: "Andes, Donald" <Donald.Andes@xxxxxxxxxx> To: <ARSCLIST@xxxxxxxxxxxxxxxx> Sent: Wednesday, May 16, 2007 5:46 PM Subject: Re: [ARSCLIST] Mass Digitization
Karl,
It seems you've addressing a few points here so I'll respond to each accordingly.
1) The subject of Audio Archeology: I don't believe there is a cohesive understanding of the state of our Recorded History and without this, it is difficult to determine the scope of the problem. In the business world, a key to understanding scope and solving problems is to apply metrics to a situation. For example, calculating what assets are lost or deteriorating, and what rate we are losing assets.
This is obviously difficult, time consuming, and comes with a lot of variables like:
-How are the loss of an intermediaries calculated? -How much deterioration is considered "deteriorating"?
Most institutions do this internally with their assets, and once determined, the next step is reviewing the loses (or potential loses) and determining if they are sustainable (which means: do nothing); concerning (which means: develop a plan, without a guarantee of funding), or vital (which means: here's blank check, get it fixed)
I can tell you that in most cases, vital is non-existent, which leaves sustainable, or concerning.
Developing concerning into actual funding takes a well developed plan that is fiscally feasible.
It's most likely that our Archeologist friends are better than we at developing cost effective plans to achieve their goal, which may be easier when justifying project costs against the collection of "priceless" artifacts. It is also very possible that we're comparing apples to oranges, as they most likely have very different funding sources.
2) The under appreciation/underpaying of Library and Archiving staff: The world today (more than ever) comes down to profitability. Since libraries don't make profits, it falls in line that there not going to be handling out high paying jobs working for a Library. High paying jobs can easily be had in the Finance, Legal, and Medical worlds. This has been true for years, but for librarians the cold hard facts haven't sunken in. Do I believe they should be paid more, of course I do. But do I think they ever will, not in my lifetime.
The fact is that we have massive amounts of history from the 1900's in every field. Are we missing important stuff, sure we are. But the unfortunate fact is that not enough people care enough about what's missing. And more so, not enough profitability can be had from collecting what was lost, to make it a worthwhile endeavor.
Think of it: That lost treasure of sound, that we thought the world would never hear again. Suddenly found, in pristine condition....How many downloads, CD's excetera could you possibly sell? Unless it the Beatles or Elvis it's most likely a lot LESS than you would think.
3) Metadata concerns: Here's the white elephant in the room. Everyone wants to preserve/transfer/digitize, but guess what??? If you don't have a complete and correct metadata standard in place, you'll probably do more harm then good. Once things are transferred, the value of storing the original drops (to the non archivist) and people assume that they'll never need to go back to it. That is until, we try to understand what the heck the file is, since your metadata seems spotty, and possibly incorrect.
4) Formatting/Migration issues: Yikes. This was hiding being the white elephant called metadata. And again, unless you figure this out UP FRONT, why bother digitizing?
5) And finally to address your last statement: I think the archiving world has it's blinders on, and needs to pull back, rationalize a bit, and find it's place in the modern world of business, technology, culture, and government. It's not effort or caring that this industry lacks; it's scope, direction and rational.
Don Andes Director of Archives EMI Music
-----Original Message----- From: Association for Recorded Sound Discussion List [mailto:ARSCLIST@xxxxxxx] On Behalf Of Karl Miller Sent: Wednesday, May 16, 2007 8:52 AM To: ARSCLIST@xxxxxxxxxxxxxxxx Subject: Re: [ARSCLIST] Mass Digitization
"Andes, Donald" <Donald.Andes@xxxxxxxxxx> wrote: ***The issue in my mind is scale because most in the archival industry are seeing a box, or room full of tapes, and have not had the opportunity to see over 1 million assets in a single location, nor contemplated what to do with them.
***If we (the archival industry) can't get a digitization schema to be cost effective, we simply won't get the funds to digitize.
***Worse, if someone outside the archival industry, gets "their" plans in motion, you can rest assure that it will not be done anywhere near correct.
***Unfortunately people don't change, and no matter how many positive reasons you give to migrate, those entrenched in analog will want to stay there.
***I believe there should be communal, parallel thinking in regards to mass digitization strategies, metadata collection and so forth. I am aware of library groups focusing specifically on metadata, but I have my own concern with their focus, and priorities in regards to collecting metadata on A/V assets.
I appreciate the perspective you bring and I agree with most of what you write.
The other day I was watching a program on the archaeological work at an early fort in the US. I noticed how many people were sifting through the layers of soil looking for fragments of pottery, arrowheads, and the like. I then thought of the estimates of analog audio in need of reformatting...by one estimate, 30Million hours. One can question if all that audio really should be reformatted, as the determination of what should survive can, even under the best of circumstances, be subjective. However, why is it that our society sees it appropriate to devote such substantive resources to archaeology while our recorded history crumbles on the shelf?
My concerns are not necessarily limited to those outside of the archival arena. I can only reflect on what I observe at my own institution. We recently advertized for an opening for someone to do reformatting. They wanted an individual conversant in Final Cut Pro, Protools and older analog audio formats. The job was advertized at minimum wage...19 hours a week...at 19 hours a week, the University would not have to pay benefits like medical, etc. Also, at our institution, it was proposed that a unique collection of orchestral performances be digitized by work study employees. Obviously, even within the profession at my institution, there is little respect given to the skills required to do the work or what it costs to pay them. Then, what were the priorities for this project? The relatively stable mylar based reel to reel tapes were the priority. Lacquer discs were not even discussed as needing reformatting.
As to the metadata concerns... I recenty read the document "Best Practice Guidelines for Digital Collections at the University of Maryland Libraries." For anyone sincerely concerned with these issues, I would recommend reading it. It is clearly a very well intentioned document, however, it seems to be have been written by those with no technical background. Their attempt to provide basic definitions is wrought with statements that I found so confusing, I was left with little sense of what they were trying to convey. It seems that we cannot even agree on definitions. And, with less than 4% of the total budgets of the ARL libraries devoted to preservation, I am left to wonder if our libraries place much significance to the preservation of our intellectual history. I am not encouraged much by what I read and observe.
While those of us who value this history work hard at changing attitudes and priorities, I wonder how we might be able to do a better job at convincing those empowered to make changes to realign priorities. Maybe our strategy needs to focus not on the inside, but on the outside. In short, I wonder, who really does place value on our recorded history.
Karl