Delivering the Next Generation Active Video Archive

Here is an article written by Jeff Boles,
analyst from The Taneja Group, Inc.

Digital video is growing in importance behind many types of different use cases today, yet nowhere is it more important than when it is captured, edited, and produced into a revenue-generating product for a video business.  

Within such organizations, video becomes the very embodiment of intellectual property – every piece of content may be viewed as a key asset that can be time and again reused or reprocessed and consequently be the source of new revenue in the form of additional final products, broadcasts, distributions, partnerships, or more. Such businesses have long been immersed in the practice of reusing digital video – typically in a cycle where a single piece of media may be recalled, redistributed, re-edited, or otherwise brought freshly back to the forefront of the business time and again.

But turning to the reuse of digital video hasn’t been a process that organizations could undertake overnight. While on the surface it seems possible to rush forward and acquire sophisticated digital capture and editing capabilities, the unwary user will quickly run into long term challenges. Those long term challenges are most often rooted in storage. Specifically, the intersection of digitized video, increasing resolutions, more frequent reuse, and sophisticated post processing capabilities have caused video sizes to burst at the seams – often pushing individual projects into terabytes of data behind the guise of one digital video. Moreover, organizations today are creating multiple resolutions of video for use in different situations – a single video asset may be transcoded at web resolutions, standard definition resolution, multiple high definition resolutions, and today, even 3D. Pure un-encoded 1080p HD video at 24bpp and 60 FPS easily exceeds 350MB/s of bandwidth. While encoded and compressed files may drop to 40 to 50 MB/s, single 2 hour long files can easily become 100GB+ monsters, and 3D video may require several times this. Moreover, video made up of randomized data, receives no benefit from enterprise de-duplication technologies common today. While it may be possible to build a system sufficient for enormous video projects, or several video projects, reusing video over time can rapidly become impossible without careful forethought.

With the challenges of both creating and then storing enormous amounts of video at the forefront, video organizations have settled on a split storage infrastructure, and rightfully so. In one tier, high speed production capture and editing takes place. In this tier, the organization invests enormous sums on performance. But in another tier – the archive – the long-term storage of video takes place. While this tier is cost optimized for storing enormous amounts of data, the media customer is well aware that this efficient repository is responsible for protecting the crown jewels of the business.

But the coexistence of these two worlds has often left a gap. Archive systems, with a focus on deep data integrity and scalability, are frequently plagued by architectures that were designed with the expectation that rapid access would never be a requirement. Storing and recalling terabyte sized projects that may span many pieces of media may create complex processes and waste hours of idle time while staff waits for the retrieval of a project. Yet the speed and performance of primary systems are bought at enormous per gigabyte costs, and scaling those systems to potentially support petabytes of mostly idle archived video is unachievable and insensible for most organizations. The media organization is stuck between a rock and a hard place. Either over invest in expensive storage that is misused for archive, or spend enormous amounts of time waiting for slow archive systems to regurgitate precious video assets as they are needed. Clearly, the mission of the media organization – storing and reusing digital video over a long period of time – would only be aided by a mix of some of the best characteristics of both production and archival storage.

The Active Video Archive
Indeed, with this in mind, we’re seeing the storage industry begin to innovate in response to the demands of media-intensive businesses. The focus for a select set of innovators rests on delivering a better video archive – one that can blend the low cost, high data integrity, and long-term retention of the traditional archive with the higher-performance of conventional disk-based storage. Such solutions can recall data to production storage with production-like efficiency, while maintaining all of the integrity, scalability, and protection a long term archive requires.

From many of these vendors, we’re seeing solutions that augment deep tape archives with disk, or better yet, deliver cost effective disk-based solutions that can still seamlessly migrate their stored data onto tape when it becomes truly inactive.  Altogether, the innovative solutions from these vendors are creating what we refer to as an ‘Active Video Archive.’ For the video customer, the benefits can be tremendous, as they turn the archive repository into a more active storage layer that can immediately serve up archival content for reuse while giving up none of the high integrity and low cost benefits of traditional archives.  

Let’s take a look at what our discussions with end users
have identified as the best in breed capabilities
for the Active Video Archive:

• Resilient – The Active Video Archive must be able to guarantee the integrity of a valuable video asset across what may be decades of reuse. Few digital storage systems are designed to store data for the period of time over which video may be reused.  Beyond high availability RAID and redundancy technology, Active Video Archival systems today go even further with deep checksumming and file segment mirroring technologies that can deliver integrity far greater than the best of traditional disk or tape subsystems. A disk-based archive can interactively leverage these advanced capabilities to constantly manage the integrity of the system – automatically policing against corruption and recopying data or otherwise correcting for data problems as they occur.
• Truly Scalable – The Active Video Archive must be able to scale in both capacity and performance to handle the expanding size and seemingly endless creation of digital assets. As experience with digital content storage dictates, it is nearly impossible to plan for tomorrow’s storage demands. In turn, video organizations should look for a storage solution that can grow beyond their most aggressive expectations. Moreover, businesses should be attentive to performance as an important dimension of storage growth – as assets increase, the number of users and systems recalling and reusing data will only grow as well. Whereas many traditional archive solutions could easily become bottlenecked when trying to access multiple pieces of media in serial, newer solutions can scale and parallelize performance by adding additional nodes to a system. Moreover, some vendors can intermix different generations of hardware and software, allowing customers to add capacity and performance without upgrading the entire archive. Such scalability over the long term can create a ‘scale forever’ single archive free from media or hardware upgrades. Some solutions can even create freedom from vendor-specific hardware dependencies through software-only offerings that can scale seamlessly across vendor-agnostic industry standard servers and hardware.
• Open and Accessible – The Active Video Archive must be easily accessed across multiple projects and systems, with speed.  Many traditional archives have not only drug down the performance of storage and retrieval with serialized access to media, but such systems have serialized access as well, by requiring specialized software for access, or requiring organizations to route all access through master servers that control and manage the only interface to the archive. Modern Digital Media Asset Management (DMAM) systems have opened up access to the archive, turning archive management into a task that takes place outside the storage repository itself. Consequently, an archive that can be accessed over open protocols, NFS or FTP for example, can allow many clients to simultaneously access an archive and reap enormous benefits in speed and efficiency.
• Affordable – With immense amounts of long-term storage at the forefront of the decision making process, the affordability of storage in a digital video archive is seldom overlooked. Today, the same disk-based systems that increase responsiveness are cost competitive with the best solutions available for archive. Moreover, cost effectiveness at scale is critical. Active Video Archives maintain their cost effectiveness, as they do not require additional entire libraries or new sets of drives to read media. Over time, disk-based solutions only maintain their cost effectiveness by realizing the benefits of constantly dropping hardware prices, and ever increasing disk capacities. By simplifying and harnessing the best capabilities of multiple types of media – tape and disk – a single well-integrated Active Digital Archive’s value proposition and flexibility can only increase.

In our view, this list of capabilities exemplifies the best of breed solutions coming to market. These innovations around digital video archival storage position such solutions to significantly enhance how organizations are working with their video assets. Not only can the best of these solutions still offer the same scalability and integrity that have always been required for long term archive, but they can do it with higher speed disk-based componentry that will convert the dusty inaccessible archive into an interactive tier of storage that is instantaneously responsive to demands for content. Moreover, the best solutions often blend disk and tape together behind a single repository that masks all the complexity of migrating data from a more performant disk tier of storage to a longer term dormant tape tier, while keeping power, performance, and capacity optimally balanced. But word to the wize, be attentive to getting the right scalability in each dimension of an archive – both tape and disk – so that the archive system is made truly scalable and unleashed from the bottlenecks of the past.

The Bottom-line Benefits
For organizations dealing with digital content, Active Video Archive innovation can have a tremendous impact on the cost of doing business, in several ways:

1. By reducing the idle time that key staff now spend waiting for slow digital video access
2. By making video content more accessible, and thereby more likely to be reused as an asset in other projects, or for additional revenue generation, and
3. By reducing administrative overhead as the Active Video Archive simplifies complex archival and retrieval processes.
4. By setting the business free from periodic hardware and media replacement that may be costly and disruptive to the organization.

The Solutions
In the leadership positions when it comes to Active Video Archives, we see vendors like Active Circle, FrontPorch Digital, Omneon, and Isilon (with select offerings) striving to provide media businesses with new opportunities to reinvent what may be painful storage practices. When turning to look at solutions for reinventing your archival content storage as an ‘active’ repository, we suggest you examine solutions with an eye toward the essential capabilities we’ve defined above. Recognize that some vendors are turning to disk as a caching layer to offset the administrative and lack-of-responsiveness issues inherent in tape, while other vendors have an end-to-end solution architected as a truly Active Video Archive. Depending on which approach a vendor has taken, a solution may be more or less scalable, and may be more or less capable in delivering the key benefits we’ve identified as having the most significant bottom-line impact in today’s rich content driven business.