Not waving but drowning in Big Data

Organisations today are experiencing a data explosion with more data being collected and stored than ever before. In fact, market research firm IDC is forecasting a 44-fold increase in data volumes between 2009 and 2020. What’s fuelling this growth? The trend toward leveraging 'Big Data' for competitive advantage means new and different types of information—website comments, pharmaceutical trial data, seismic exploration results, to name just a few—is now being collected and sifted through for insight and answers.

At the same time, organisations are capturing and converting more and more content into digital data. Data types, such as video, which used to be considered a ‘nice-to-have’ on a company’s website, have become essential marketing, training and communications tools. Advances such as special effects and high definition significantly increase the amount of data generated; it takes twice as much space to store 3-D video as 2-D video because the technique requires two cameras to shoot the same footage, putting a significant strain on storage resources.

This was exactly the challenge facing Park Road Production, a post-production facility for filmmakers located in Wellington, New Zealand. Park Road has scaled its systems in recent years to accommodate the rapid increase in data that digital film-making now generates. As production of digitally acquired 3D films is becoming more commonplace, uncompressed digital film can translate into hundreds of terabytes of data per project, and the traditional tools used to manage the content are often overwhelmed. In short, Park Road was faced with the prospect of “drowning in data.”

Attempting to support such advances in technology can reveal significant gaps in IT architectures. Also driving up the amount of data that companies must store today are the growing obligations that regulations and laws are placing on companies gathering and storing data about customers, partners and even employees.

As businesses, non-profits and governments realise the importance of data—not only for day-to-day functions, but also to their strategies and success going forward—IT departments face rising challenges in developing strategies to support the organisational dependence on data. These include:

• Budget constraints. In this uncertain economy, organisations want to make the most of what they already have, not to commit significant up-front investments to new technology.
• Complexity sprawl. With technology talent also strained, IT organisations are loath to add more complexity to their technology infrastructures and are instead looking for more ways to empower end users with easy-to-use tools.
• Availability requirements. Companies need to ensure their employees can get at their data easily, regardless of where it may be stored or how old it is.
• Integrity concerns. Companies expect the data they trust to storage will remain intact and unaltered. IT departments need to be able to promise such capabilities, and keep those promises.

An eye towards archiving

Setting strategies to archive data in an automated, reliable, cost-effective way is emerging as the right approach for organisations in a wide variety of industries. Archiving older data so that it’s automatically moved to less expensive storage media—such as cheaper disk or tape—frees up costly primary storage. That means as companies collect more data they can prioritise it based on how old it is and how often it is accessed. When done properly, archived data is inexpensively but reliably stored and easily accessed.

However, in today’s complex world of storage and data management, archiving is not always given adequate consideration—or worse yet, not done at all. Archiving data is often confused with backing up data, as companies assume they can do one or the other to cover their bases. However, companies that simply back up data are wasting expensive storage equipment and tying up precious IT resources by keeping all data on a primary storage tier, regardless of the age of the data or how frequently it is accessed.

What’s more, these companies are missing the big picture by failing to think about their long-term needs, including which type of storage is best suited for their data over the years. Most companies perform backups, but not as many companies today are truly archiving their data. Industries like media and entertainment, which are dealing with such massive amounts of digital data that they just can’t keep it all on disk, are typically ahead of the broader market.

Let’s take Park Road as an example. The post-production company now has a solution in place that allows it to automatically move data between high-performance disk and a large capacity tape library archive. The solution also provides a substantial increase in the amount of data that can be kept for near-term re-evaluation or processing, as well as long-term archive. The system at Park Road routinely processes multiple terabytes of data in just a matter of hours and can handle in excess of 20TB per day at peak load.

Archiving advantages

More generally, well-planned archiving strategies can provide organisations with a number of benefits:

• Cost savings: Archiving systems enable the movement of data that hasn’t been accessed in a certain amount of time—usually defined by the IT department—from expensive disk-based primary storage to less expensive second-tier or third-tier storage, either disk-based or tape. This frees up primary storage space for the ever-growing amount of data organisations collect on a daily basis.
• Reduced management: Unlike backup strategies that require users to request restores from the IT department, comprehensive archiving strategies allow end users to find needed files themselves. Archiving solutions that include file management software make this task simpler by giving end users a Windows-like file management system that’s easily navigated, so they can find and access their files in their original format.
• The right storage media for the right data: Organisations need fast access to data that is called up often. To suit these performance needs, primary, disk-based storage is the right fit. However, as data gets older and is accessed less often, companies can reap significant cost advantages by moving older data to lower-end disk or tape. While these solutions don’t provide the fast access times that high-speed disk does, such performance isn’t required for older data that isn’t accessed for long periods of time.
• Assurance of data availability: Archived data is often kept for years, so an important element of a comprehensive archiving strategy is a solution that enables information stored for the long term to be available through the years. Solutions that regularly check the health of storage media provide assurance that data stored for the long term remains intact.

Whether it’s the onslaught of 'Big Data', the prevalence of new types of digital data in corporate environments or burdensome compliance requirements, organisations of all sizes are pressed to deal with the data explosion. Strategic, comprehensive archiving solutions present affordable options that save IT resources, reduce strain on IT personnel and can scale with a company’s needs.

Michael Shing is Quantum's ANZ Big Data specialist.