RE: [AV Media Matters] Toward a MediaLESS Archive?

[Jim Lindner <jim@vidipax.com>]

Thank you Eddie for an excellent posting -

A few comments:
1. My current thinking is in many respects CONTRA the existing
discussions that you mention.  The notion of monolithic deep storage
sites is one that I dislike the more I really examine it.  One of the
basic ideas that I am grappling with is the idea there is really no
single repository where the AV Object lives. Quite the opposite of a
data warehouse.  I thought I made that clear - but to elaborate.  I am
thinking of a bitwise distribution here which is a notion that I have
not seen discussed previously.  The idea is that the content is
digitally sliced and diced so that NO ONE has a single copy in storage
in an integrated form.  So in any given byte - bit 1 and 3 may go to
storage vendor 1, bit 2 and 4 to storage vendor 2, bit 5 and 7 to
storage vendor 3 and bits 6 and 8 go to vendor 4.  So each vendor has
only a very small portion of the work.  The more vendors the better -
the I/O is split among many different providers and bottlenecks are
reduced. Further_
2. The work does NOT reside for a long period of time in any of the
vendors.  I suggest a digital storage barter "demon" that sits on the
network of the content owner and does inquiry into the costs of storage
at any given vendor site.  If one views storage as a commodity, then one
can trade it as such (in fact one could even concieve of a "futures"
market in storage - but that is getting a bit too off topic!).  So the
notion is for a free market trade of storage as a commodity - and the
"demon" at the users location just sits there 24 hours a day and sees
who is selling the storage at what price and then buys and sells storage
according to the market.  Then - the demon moves the materials in a
bitwise way from one vendor to another.  So - the content is NEVER in
stasis, rather it is always moving from site to site, as bits - not full
content.  The concept of a FLUID archive is directly opposite to the
ideas I have heard before of monolithic or even distributed data
warehouses where the data representing the content is stored in deep
storage - which if you think about it is really more or less doing what
we are doing right now - albeit in a digital form (which has some
advantages - agreed) but still has many of the same vulnerabilities. I
know it is a bit of a hard idea to imagine - but the content is almost
always in transit - and that is a very powerful idea.  One can view it
from many different perspectives - but I do mean mediaLESS in many
respects.  Yes, data in fact MAY reside for short periods of time at any
one vendors and can be stored on - disk, tape, neurons , whatever - but
it is only there for very short time periods because the strength of the
idea is that the data is always moving. So -
3. A key element is using RAID not as a specific scheme for disk drives
- rather as a "philosophy" that ties it all together.  If things are
distributed in a BITWISE way - what happens if you lose some of it.  The
solution is that you engineer the system PLANNING for it to lose it so
that the system is totally fault tolerant - and the reason it is, is
because the actual content really resides in MANY places, and is moving
all the time. If for a moment we look at vendor one that is our
MOMENTARY custodian of bits 1 and 3 of a particular byte of a particular
string of files - and we predict that on occassion we will lose the
information - can we develop a scheme so that we can recover - the
answer is yes - we do it all the time.  So the bitwise stream is error
corrected as a matter of course - it is planned for.  What does it
matter if vendor 1 loses it's entire inventory - it really does not -
other then the issue of time that will be required to reconstruct the
lost data - and we can probably do that in REAL time.

I hope that I have clarified this somewhat - this is definately a
"thought in progress" and I very much appreciate the input.

jim