Branch office WAN optimization has been a top project for IT professionals the last several years and has created some new IT vendors in the process. New initiatives like large disaster recovery replication, regional VMotion and now big data analytics are driving a demand for data center-to-data center WAN optimization.

Traffic between the central data center and remote or smaller offices is often called “north-south” traffic and typically involves latency between the connections. This latency means lower transfer speeds which allows for the use of less efficient WAN appliances to keep costs down. Data center-to-data center WAN optimization, on the other hand, doesn’t have the latencies common in branch office optimization.

This “east-west” traffic between data centers is often very high-speed ranging from 10 Gb per second to 100 Gb per second. This means the WAN optimization solutions designed for north-south applications may not have the performance necessary for this inter-data center traffic.

Infineta’s Data Mobility Switch is designed for this ‘big traffic’ requirement and can provide 5 to 10 times the bandwidth capacity, IP traffic acceleration and application prioritization, compared to branch office WAN optimization solutions. For inter-data center workflows like production applications, VMotion over distance, large file transfers and now big data, this type of high performance is a necessity. Recently, storage Switzerland discussed with Infineta the impact that big data IT projects are having on the WAN.

Big data, big WAN

Big data initiatives are key focus areas for large organizations today. While many companies are still in investigative or proof of concept phases, there are companies that have moved into production and are seeing significant returns on investment. With these successes come certain problems. As we discussed in a recent article “Designing Big Data Storage Infrastructures” these environments can grow to petabytes in size. But for maximum value the data needs to be disseminated throughout the organization. That dissemination happens via the corporate WAN, which puts tremendous pressure on the network infrastructure.

The first pressure point for the WAN is data aggregation, which involves moving data from a collection point or points to a centralized analytics cluster, often based on Hadoop. Next the cluster aggregates that data and begins processing. Often, this processing is not limited to a single Hadoop cluster or even a single data center. It’s shifted between data centers based on processing capabilities or a desire for the results to be generated closer to where they’re needed. There is also demand on the WAN for DR replication of this big data since, in many cases, the data being collected cannot be re-created. Protecting it and making sure it’s copied is critical.

Finally, the WAN is needed for results distribution. Big data will often generate subsets of data that will be further processed and may even be combined with other structured data warehouses for further comparison. In these cases it’s ideal for that data to be moved closer to where the next round of processing will be done or the results reviewed.

Big data is not just another disruptive technology. It’s a process by which IT can provide information assets to the organization that can be mined for decision support and better  profitability. In short, big data, unlike many other IT initiatives, can make organizations money.

The distribution of data is a critical requirement for a successful big data project. That means more is going to be demanded of the WAN. The key though is for big data’s requirements of the storage and WAN infrastructure to be minimized so that the cost of that supporting infrastructure doesn’t outweigh the benefits that big data can deliver.

Being able to provide fully optimized WAN connectivity is going to require products that are focused on that task and have the ability to optimize WAN traffic without impacting the low latency connections between data centers. Companies like Infineta with their Data Mobility Switch are well-positioned to take advantage of this opportunity and allow customers to further exploit the revenue generation potential of big data.

Infineta is not a client of Storage Switzerland

 

previous entry: “Top Storage Companies To Watch in 2012”