Problem


Root cause analysis in complex systems like FC SANs is difficult under the best of circumstances. The lack of information about real time data transactions that most network monitoring tools provide can make fixing a performance problem even harder. This can lead to the practice of replacing system components, one after another, in the hopes of solving the problem. So, if the driver for an upgrade to 8Gb is a need for better application performance, and network bandwidth isn’t the bottleneck, that upgrade may produce some disappointment.



Risk of no improvement


Investments in infrastructure are expected to bring a return, like improved performance, and can include a very real risk if these expectations are not met. Aside from the personal and political aspects of making a bad decision, spending resources on an 8Gb network to solve a performance problem that it doesn’t address still leaves you with that problem to solve and potentially less resources to do it with. In general, overspending on infrastructure upgrades also carries the risk of increasing opportunity cost, essentially consuming resources that could have been applied to another problem.


Similar to the ‘no improvement’ risk is the risk of moving the bottleneck to another part of the system. If increasing network speeds to 8Gb just causes other components to become the bottleneck, this upgrade can force additional unexpected expenditures (maybe faster storage, like SSD to utilize the 8Gb network) fueling the ‘upgrade cycle’ and adding to the opportunity costs.



8Gb is more demanding


While it’s good to be proactive, it’s not good to invite problems. Each increase in network speed reduces the bit period, or the ‘window’ for valid data transmission, proportionately. Each reduction in bit period requires a more robust physical network infrastructure like optical cables, connections and components in order to prevent an increase in transmission faults and bit errors, which can lower effective performance or reduce reliability. In other words, faster networks have reduced ‘margins for error’ which can make them harder to support and increase the administration and troubleshooting workload required to maintain them.


Replacing 2 or 4Gb components, or adding 8Gb components to a stable, functioning network and increasing network speeds can bring the risk of reduced reliability. Advancing the speed of an existing network simply because that extra speed is available can invite unnecessary risks.



Solution - Instrument the infrastructure first


As a best practice, it’s advisable for larger enterprise data center managers to install Fibre Channel network TAPs and monitoring software like VirtualWisdom by Virtual Instruments to generate real time, granular transaction data before the decision is made to add 8Gb components to the SAN. Other devices like light meters and protocol analyzers can be used to monitor signal quality and energy levels and indicate the network’s ability to handle faster transmission speeds. For complete optimization of higher bandwidth, it’s essential to capture every SCSI transaction between servers and storage volumes for every network link that would carry critical data. As passive, out of band components, TAPs transparently collect a copy of all data that’s flowing through the SAN and send it to the monitoring software.



Monitor infrastructure to determine readiness for 8Gb


As mentioned, each increase in transmission speed decreases the bit period by the same percentage, meaning an 8Gb network is twice as susceptible to jitter and transmission errors as the 4Gb network it replaced, and 4x as apt to have problems as a 2Gb SAN. After instrumenting the network with TAPs, an optical power budget and error analysis can be made to determine if the existing infrastructure is ready to support this speed increase. Dirty connections, lower-quality SFPs, excessive cable lengths or tight bends can reduce the power margin, or the net signal strength after power loss from cabling and connections. It may turn out that transceivers need to be upgraded, cable lengths shortened or routing modified to reduce light loss in a network segment.



What to upgrade


With the network physically ready to accept 8Gb traffic, the next step is to identify which components are causing bottlenecks and need to be upgraded first, and those that aren’t and may not need to be upgraded at all. The accurate, real time data that an instrumented network provides can prevent the ‘rip and replace’ upgrade practice that many organizations fall into when they can’t identify the root cause of a performance problem. This can, in turn, prevent a succession of unnecessary component replacements that don’t affect network performance, but can affect job performance.


As is often the case, poor application performance may not be a result of an over-taxed network. In fact, due to a lack of visibility into real time SAN performance, most networks are actually highly underutilized, often producing a waste of resources that storage managers typically receive the blame for. Based on the experiences of companies in this industry, like Virtual Instruments, helping customers maximize SAN performance, it’s reasonable to estimate that data centers with 2 and 4Gb FC SANs typically have utilization rates in the low single digits. While installing TAPs and monitoring software to gather real time transaction data can show which components or network segments could benefit from being upgraded, it can also show if a network upgrade is not needed at the current time. In the final analysis this may be the appropriate (and least risky) path, postponing the decision to upgrade the speed of a SAN or given network segment. This can include electing to run new 8Gb components at 4Gb speeds until the rest of the network is ready and the increase is warranted.



When to upgrade


In this case the investment to install the TAPs and network monitoring systems can be recovered quickly, if it justifies a decision to not upgrade to 8Gb, or even delays that upgrade by as much as a year, which is not uncommon. And, when the time comes to do the upgrade, this instrumentation will be needed to determine where and how to implement it. When the switch to 8Gb components in the SAN begins, it’s considered a best practice to run them at 4Gb speeds until the infrastructure is completely ready and stable, another decision made easier with TAPs and SAN instrumentation. In the long run, on-going, real time monitoring can support easier day-to-day operation, and ensure full utilization of this higher performing infrastructure. Finally, the upgraded management capability this monitoring system provides can reduce the risk of increased admin and support time consumed by a more demanding 8Gb network.


Increasing network speeds to support 8Gb FC components can carry some risk. And the same will be true for 16Gb FC solutions when they are commonly available in a few years. Given the complex nature of networking infrastructures, determining root cause for poor application performance is difficult and, more often than not, the problem isn’t network bandwidth. When upgrades are driven by the need for better performance, there’s a very real risk that the decision to upgrade will turn out to be a poor one. Also, each increase in network speed brings potential reliability problems, as the margin for transmission errors goes up proportionately. Before a decision is made to upgrade, instrumentation of the FC network is needed to provide real time information on data ‘transaction fitness’ of the SAN. Once in place, this monitoring infrastructure can provide the data to decide which network segments to address and when and reduce these potential performance and availability risks associated with an upgrade to 8Gb fibre channel.

Eric Slack, Senior Analyst

Virtual Instruments is a client of Storage Switzerland