The Petabyte Problem

Media and Entertainment has a unique situation. This industry has been affected by technology maybe more so than any other because of these fundamental shifts in the industry’s pipeline:

There are two different film workflows using storage in slightly different ways:

Traditional film with digital effects, often scanned from celluloid at 2K or 4K resolution

The production of animated features

Increasing storage requirements are being driven by:

The Drive towards higher quality (enabled by key technologies such as scanners and editing environments that handle high quality)

The complexity and number of effects in a feature

3. Increasingly parallel production (working on 3 projects at once instead of one at a time)

The growing adoption of all-digital HD and film workflows for post-production and visual FX studios has enabled the industry to produce more complex and more visually inspiring scenes. The sum of these parts suggests that the Media and Entertainment industry is living in a Petabyte World, a world that is getting bigger and bigger everyday.

The problem that studios are facing is that although many effects and scene constituents can be reused to improve creative efficiency, it is very hard to a) find and b) restore this data for reuse. The ability to efficiently access and reuse of scene data is driving the desire for nearline archival of film data and a move away from tape-based archival.

The storage and network infrastructure has the ability to keep pace with the demands placed on it by high-speed film scanners, direct digital acquisition cameras and fast multi-core computer processors and buses that are manipulating large images in real time. The challenge is the budget does not have the ability to keep pace with the amount of storage required to sustain these projects. As a result only short segments of a digital media project could be stored in online storage, and older projects could only be stored in tapes on a shelf.

The Studio Systems Administrator

Despite the need to manage storage issues like provisioning storage for new projects, protecting the digital assets, maintaining a disaster recovery strategy and organizing data archives, the studio’s system administrators (SSA) have the added responsibility of having to manage the project's pipeline as it progresses in terms of maintaining storage capacity requirements for that project. Properly maintaining storage capacity for a given project let alone multiple projects forces the SSA to become an acrobat, balancing the demands of speeding the project along vs. the realties of storage costs.

Solutions to Managing Digital Media Storage

When trying to solve this problem there are two common solutions that are often attempted -- keep all the data cold or keep all the data hot. What is needed and is now being made reality by companies like Ocarina Networks is keeping all the data warm.

George Crump, Senior Analyst

for managing Media and Entertainment Data


Keep it Cold

The current solution for most large studios has been moving inactive sections of the project to tape. Albeit very cost effective from a physical media standpoint, it is very costly from a time perspective. It also involves the studio systems administrator being a go-between and potentially a bottleneck should a creative engineer need to edit a scene or file that has already been moved to tape. This results in a time-consuming effort whereby the administrator is essentially looking for a needle in a digital haystack of archives.

The cold strategy for many organizations involves, as sections of projects become static, moving those sections to tape for less expensive storage. This has many negative project delaying ramifications. Almost always a section of the project needs to be recalled from tape for reuse or touch up. Sometimes this is a request that can be planned for as the project hits the final stages or sometimes it is an emergency rework that may stop the project altogether until the situation is resolved.

In either case this leads to the normal challenge of time and process involved in any tape based recovery. The tape that has section of the project on it has to be identified. Compounding this is that the creative talent involved in digital media may not be able to communicate the exact components of the project that need to be restored. It is not as simple as a "restore file x" type of request. In some cases multiple tapes have to be requested and scanned to find the right sequences and source data.

Once the tape is identified, it has to be sent back to the data center, loaded into tape drives and scanned to the appropriate location for the needed data sequence, and restored. The first step in restoration, especially relevant in digital media, is finding space on disk to restore it to. This ironically may force the premature migration of a near hot part of the project, which in turn could lead to another unintended recall from tape request.

Once the space on disk is identified, the project section needs to be recovered. Again, significant in digital media because of the size of the recovery. In almost every case this needs to be restored across a network to the NAS storage that houses the projects. It can take hours in some cases for just this transfer to take place.

This process thus far has assumed nothing goes wrong. What if something does? Tapes in particular are prone to failure, but networks can fail, cause errors or corruptions, they can also become constrained by the restore affecting other users of the project.

In an industry where time-is-money and money-is-time, disruptions to the project pipeline like this can be very costly. The alternative is to keep everything online and hot.

Keep it Hot

The seemingly ideal solution would be to keep all the data online for the life of the project on high performance tier one storage. In a Petabyte World like digital media, the storage budget would seemingly have to increase as much as the data growth. Despite the decrease in storage capacity costs, and especially in today's economy, this is not a realistic solution.

The good news is that high performance is really being driven by the demands of the render-farm (many linux nodes asking for the same content) and handling raw video in the edit suite (retrieving large sections of video with real-time edit interactivity). The real ideal solution then is to move anything else can to lower-cost storage. The problem is that the studio systems administrator has struggled with how to deliver this balance.

Most studios after examining the Keep it Hot strategy begin to look for something in the middle; the ability to keep it warm.

Keep it Warm

The keep it warm strategy essentially involves using less expensive disk to store inactive sections of projects. The attempt is to strike a balance between the slow recovery of a cold strategy and the unsustainable expense of a keep it hot strategy. While the keep it warm strategy greatly improves the recovery performance of a tape-based strategy, it does have some inherent problems.

First the data still needs to be manually managed and recalled when requested. As a result the studio systems administrator is still the digital media librarian and can still be the bottleneck. Second for some semi-active pieces of the project, the time to copy the data from the secondary storage to primary can still cause a performance problem. As a result the SSA cannot be as aggressive with migrating static sections of the project as they might like to be.

Finally the cost delta between primary storage, secondary storage and tape storage may not be significant enough. Tape may still need to be actively used because even the cost associated with secondary disk storage may be too high. Not only does the warm strategy then still have all the recovery challenges associated with the cold strategy it all has the added administrative burden of managing three tiers of storage to the SSA's already full plate.

Keep it Simmering

The keep it simmering strategy is a relatively new one that leverages storage optimization technologies like those offered by Ocarina Networks and it’s ability to interact with both primary and secondary storage.

First through content aware optimization storage capacity requirements on all but the most active data can be reduced by as much as 50% without impacting the storage system’s ability to maintain high performance on that most active data set. In the digital media data center capacities of 100TB's plus are not uncommon, imagine reducing that to 50TB's and the savings are substantial.

This optimization is done through a combination of compression and deduplication. Inactive data is identified based on rules set by the SSA. The digital media data is then extracted by the content aware optimization software from its original format. It then delayers and decodes the file into sub-file component objects. Once the underlying objects are identified they are then correlated into common object within the same file and across other files, this eliminates the need to find exact file-level matches for de-duplication. Finally the data is then compressed by digital media specific optimizers for maximum reduction and compacted into containers for maximum on-disk efficiency.

The next advantage of the keep it simmering strategy is to then manage the location of the optimized data. The data can be optimized and left in place. This eliminates the need to move large data sets back and forth across the network and as a result allows the SSA to be very aggressive in what project sequences get optimized. If the data is in place it simply needs to be read out of its current optimized state and into its original state. This re-hydration is done automatically without SSA involvement and little impact on performance. This allows for optimal efficiency of the primary store with minimal impact on workflow.

Based on rules set up by the administrator optimized files can also be moved to secondary storage. Once the rule is set this is an automatic action and again requires no intervention by the SAA. This not only solves the problem of knowing what to move and when, it also further enhances the price delta between primary and secondary storage, making that secondary storage even more cost effective.

Most importantly the movement of the secondary data back to primary is automatic and initiated by the creative team. They simply see their existing storage hierarchy, unaware that optimization has taken place and that the actual storage location of the data may have changed. This then eliminates the SAA interaction with the creative team and allows for more aggressive migration resulting in additional cost savings.

Finally even the move to tape can be optimized. While tape would in this strategy be relegated to a DR function, the data can be stored to tape in its optimized format. This allows for a significant reduction in the number of tapes that need to be managed and recalled in the event of a full system or site failure. If the number of tape mounts can be cut in half and those tapes can restore twice as much data (because it is optimized) in the same amount of space, the overall recovery process can be greatly sped up when it is needed most because the business is down.

The keep it simmering strategy enables SSAs to leverage storage to bring digital media projects in on time and under budget by recovering useable space on primary storage already in place.  This also provides help with migrating data so that it’s seamless to the studio animator and as a result doesn’t have to involve the SSA so it reduces the resources of bringing things back from tape as well as wasting time looking for files.

The bottom line is the keep it simmering strategy reduces storage expenditures while increasing administrator and creative talent productivity.