As data usage continues to grow exponentially, IT managers will need to
orchestrate multiple kinds of storage — including flash, hard disk, tape and cloud
— in a way that not only optimises capacity and performance, but cost and
power consumption too. Storage costs are continually dropping in the fields of
digital repository – and yet remain high because we are simply producing more
of it.
When I got my first computer back in the… oh a long time ago, I remember
wondering how I could possibly fill all 20 megabytes (MB) of hard drive space.
Last week I installed a trio of 4 terabyte (TB) drives in my Mac, wondering if the
extra space would last me through to the end of the year! Due to the digital
revolution that we are all caught up in, the need for high-capacity storage is
evident more than ever before and more importantly at low, affordable cost.
In an effort to put it all into perspective I found some historical pricing data to
illustrate how the cost of storing a gigabyte has changed over the last 30 years.
Of course a gigabyte was a laughable concept to all back in the 80s, but it’s a
nice middle-of-the-road magnitude for the purpose of this discussion.
In 1980 Morrow Designs released their first 26MB drive (a huge jump over
Apple’s 5MB capacity) at the incredible cost of just US$5 000. In 2014, Hitachi’s
HGST 4 TB (4 000 000 MB) Deskstar hard drive costs just $179. Comparatively;
1GB of data in 1980 would have cost US$193 000 to store, today the cost is just
US$0.044 per gigabyte.
More data
In today’s world we are obviously creating more data than before and as we
become increasingly dependent on digital data, so its storage, management,
transport and governance become ever more critical. The estimated size of the
digital universe (that is, all the digital data created, replicated and consumed) in
December last year was said to be 2 837 exabytes (EB) and it was forecast to
grow to 40 000EB by 2020 – roughly doubling every two years. So by 2020, the
digital universe will amount to over 5 200GB per person on the planet.
This sort of data growth clearly requires heroic amounts of storage to keep it
available for as long as it’s required, and to archive much of it thereafter. And
herein lies the problem: although the cost of hardware has dramatically reduced
over the years there are major capacity limitations for the amount we are
producing.
The Academy of Motion Picture Arts and Sciences archives over 70 000 movies.
They can do this efficiently and inexpensively for one reason: Photochemical film
is cheap and easy to preserve. All you need is a cold room that’s not too humid
and not too dry, and the chemically processed film will last for 100 years or
longer.
Digitised movies are not nearly as easy to archive for the long term as good old
film. Apart from the requirement of preserving an 8.3-terabyte digital master, a
major Hollywood production racks up about 2 petabytes (PB) of ancillary data
(the equivalent of almost half a million DVDs). Where do you store 2PB when,
worldwide, we are producing about 7 000 new feature films each year? The
uncertain state of digital preservation is such that most major studios continue
to archive their movies by transferring them to separate, black-and-white
polyester-based film negatives, one each for red, green, and blue, even for
those works that are born digital.
New technology
Although hard drive technology is getting bigger and cheaper by the year it still
does not provide the capacity and is not 100% reliable over the long term.
Having said that, Hitachi is developing so called “cold storage hard drives’,
Helium filled drives that offer greater capacity, less drag on the spinning platters
and therefore, longer serviceability.
LTO technology has proved to be the cheapest and more stable format for data
archiving but again we are fairly limited capacity-wise. A breakthrough in tape
technology was announced by Sony at the beginning of May this year however.
Sony has succeeded in creating a nano-grained magnetic layer tape able to
record approximately 74 times (185TB) more data than conventional magnetic
tape media (LTO 6s 2.5TB) at a supposedly lower cost and this may be a step in
the right direction for our mass storage needs.
For large collections of important or valuable digital data – not just motion
pictures but news footage, scientific measurements, governmental records, and
our own personal collections – today’s digital storage technologies simply do not
work to ensure their survival for future generations. So if you’re an engineer or
computer scientist or just someone like me up for a grand challenge, investigate
ways to make the problem of digital information decay become obsolete and add
digits to your bank account rather than to another hard drive!
