Preserving the born-digital: "The future for digital storage is constant migration."

Since I'm not really done with the previous post, but think this deserves a dedicated one to itself, the article I linked to, on the question of preserving films that were 'born-digital':

The preservation of born-digital films is going to be the greatest challenge ever to face archivists.

Margaret Bodde, Executive Director of the Film Foundation

The new magical software has sometimes led to over-restoration. Grain has too often been polished out, creating a plastic sheen. Still, today no archivist can avoid using the new toolkit. The sadder story involves not restoration but conservation and preservation. A civilian might think: That’s simple. Just save film on film and digital on digital. But things are more complicated than that.

Let’s start with a movie like The Girl with the Dragon Tattoo, which was shot with digital capture. After production and post-production, it was made available to theatres as both a Digital Cinema Package (that batch of files on a hard drive) and some 35mm prints. But there are several digital versions of the movie.

The Digital Source Master: This is the original sound and image “content” captured in specific formats, either tape-based or file-based, and those may come in many flavors. The Girl was shot with the Red One camera on the company’s proprietary format R3D. That material, along with sound recording, was converted to other files in postproduction. Any major film nowadays is likely to use many digital video and audio formats. This entire set of materials forms the Digital Source Master for a film and these assets are usually stored in the studio’s vaults. Along with them are, usually, film-based copies of the final product, often as separation masters.

The Digital Cinema Distribution Master, in standards specified by the Digital Cinema Initiatives. This is the finished film unencrypted and uncompressed, providing “content” at 2K and/or 4K resolution. Roughly speaking, this is the digital counterpart of a 35mm film negative.

The DCP, compressed and encrypted for theatrical playback. It is, again in some respects, the digital counterpart of an analog film print.

Eventually, The Girl will show up on the optical disc formats DVD and Blu-ray, not to mention streaming video, cable transmission, and web-based platforms. (Actually, it’s probably already available for Darknet download.)

Many studio films are housed in nonprofit archives too, and until recently those movies have been deposited and stored as analogue copies. But what will those institutions now keep? There are only three minimally acceptable formats: the uncompressed and unencrypted DCDM, the DCP, and a 35mm print. Suppose your film archive is lucky enough to receive both a DCP and a 35mm print of The Girl.

First, how do you access the DCP files? A DCP is typically encrypted to block piracy. When The Girl played theatres digitally, each exhibitor was provided an alphanumeric password that would open the files for loading into the theatre’s server. By the time you the archivist get the files, that key may have expired or been lost. Without the key, the DCP is useless.

Then there’s the matter of storage. The 35 print of The Girl can simply be passively conserved, following the motto, “Store and ignore.” But all digital material, no matter how minor, requires proactive preservation. The future for digital storage is constant migration.

Archivists estimate the life of any digital platform to be less than ten years, sometimes less than five. All hard drives fail sooner or later, and they need to be run periodically to lubricate themselves. Tape degradation can be quite quick; one expert found that 40 % of tapes from digital intermediate houses had missing frames or corrupted data. Most of the tapes were only nine months old.

Moreover, hardware and software are constantly changing. One archivist estimates that over one hundred video playback systems have come and gone over the last sixty years. Archives currently recognize over two dozen video formats and over a dozen audio ones.

Periodically, then, the DCP files of The Girl will have to be checked for corruption and transferred to another tape or hard drive and eventually to another digital format. Such maintenance takes time; shifting a terabyte of data from one system to another may need at least three or four hours. Ideally, you’d want several copies for backup, and you’d want to store them in different locations.

There are hundreds of other films like The Girl awaiting processing at major archives. About 600-900 features are produced in the US each year. Currently the world is producing about 5500 films per year. At some point, they will all originate in digital capture.

Besides access and storage there’s the matter of cost. Storing 4K digital masters costs about 11 times as much as storing a film master. You can store the digital master for about $12,000 per year, while the film master averages about $1,100.

How do the overall costs of digitizing mount up? Look at the situation in Europe. The EU countries produce about 1100 features and 1400 shorts per year. An EU archival commission, the Digital Agenda for European Film Heritage, estimates that to conserve one year’s output would require 5.8 PB (petabytes) of storage. In 2015, the costs of archiving that year’s output (without restoration) are projected to be between 1.5 million and 3 million euros. Beyond initial conservation, long-term preservation of that year’s output would consume, though migration and backing up, about 1900 PB and cost about 290 million euros.

The access problem is soluble. Your archive could be given an unencrypted DCP of The Girl and then create its own key to prevent copying. Or the DCP could be assigned a generic key, perhaps for a specified time period, that will open the files in a secure milieu. They could then be migrated to an format under archive control. On the matter of software, archivists are working on establishing standard preservation file formats and codecs. To deal with the other problems, you’d have to press for increased budgets and personnel to cover the new duties that digital archiving creates. But the costs, including training personnel on ever-changing platforms, are of tidal-wave proportions.

So there you are: first there's the ease of technology and then - as any editor will tell you - the sheer migraine it induces.

**

Not entirely related, but this: "Once you’ve found a way to conserve-preserve The Girl with the Dragon Tattoo, what if you want to show it tomorrow? Or ten years from now? Or fifty?"

I feel they could have chosen another film as an example, because this just....makes me giggle.