The storage I/O challenge is being created by mechanical disk drives not being able to keep pace with Moore’s law where CPU processing power effectively doubles almost every year. Networks, while they lag Moore’s Law, have kept much better pace especially when you factor in the currently underway upgrade to 10GbE and 8GB FC. As stated earlier disk drives have been stuck at 15K RPM since the late 90’s. Until very recently storage system manufacturers have traditionally worked around this problem by using shelves and shelves of disk drives to get a combined spindle count that could keep pace with the storage I/O demand. This of course lead to more capacity being sold than is needed and explains the persistent utilization surveys that show less than 30% of total storage capacity is typically used in a data center. With the price drop in solid state technology and this wasted capacity many data centers have began to explore the technology as a viable performance option.


Storage systems manufacturers were not prepared to cede the high performance storage market and as a result a growing number of storage system manufacturers have begun to use solid state to try to address I/O demands in the data center. The problem though is to move to the technology quickly, legacy storage vendors adopted the use of solid state that was designed to fit into the same slot as a mechanical hard disk drive (HDD), hence the SSD. While these systems may be fine in a laptop or single server, when placed into a shelf that was designed for legacy systems with today’s HDD aggregation technologies, solid state cannot reach its full potential. The limitation of the legacy storage system creates artificial limitations on the SSD. The legacy shelf can’t support the I/O capability of a combined group of SSDs. As a result the legacy shelf can’t be fully populated with SSDs and some drive bays (50% or more) must go paid for but unused. Even if the shelf could support the SSD’s speed, they are designed to provide room for the vibration and cooling needs of mechanical drives. Far more SSDs could be installed in the same space that a legacy disk shelf occupies.


Beyond these artificial limitations SSDs also face real limitations when solid state memory and the Flash controller technology is confined to the space used by a HDD. The HDD form factor limits how much Flash and corresponding memory cells can be implemented per drive. One example is the performance impact over time. Performance of a Flash memory device changes as data is written to it. Any Flash device will operate its best performance until it becomes filled with data and the controller software needs to reclaim retired blocks. If the page has old data then the entire block needs to be erased before the pages can be written to. The more of these "erase then write" cycles that have to occur the more performance is impacted and dramatic latency spikes will be experienced. SSDs are then encumbered by a write cliff effect where write performance drops off after all the free Flash devices have been initially written to and the device cannot provide enough free blocks to keep up with the write requests. The less Flash capacity a system has the sooner this occurs and the more significant the drop off. In the 7x24x365 enterprise data center there is rarely a “quiet time” for the SSDs to perform their housekeeping duties (reclaiming used blocks) while not having to process critical data at the same time.


While it is true that SSDs in legacy storage arrays do improve performance compared to the mechanical drive alternative, the problem and the reason enterprise adoption is limited is the performance capabilities are squandered compared to the actual potential of Flash memory. Coupled with the lack of Flash density that legacy storage systems provide there is an inability to strike a reasonable balance between cost and performance. This has lead to the continued adoption of high drive 15k RPM mechanical drive based arrays. The alternative is the Flash Memory Array.



What is the Memory Array?


A Memory Array aggregates Flash memory components using a hot swappable DIMM form factor, and placing many of these modules into an array designed from the ground up to take advantage of the capabilities of Flash. The result is that the SSD, at least as they are used in legacy enterprise arrays, are similar to the SCSI to fibre channel conversion devices we saw when fibre channel storage networks first came into being. As the world became fibre only the need for such conversion devices no longer existed. The same may happen with solid state storage, as memory becomes increasingly the medium of choice much of the technology used today to retrofit it into the environment will no longer be needed.


By aggregating Flash into an array like design that is not encumbered by having to also support the mechanical hard drive has several inherent benefits. First, it is a systems level approach to designing specifically to accommodate only Flash memory and not to retrofit the solution into a legacy storage system. The primary advantage of this type of design choice is density, serviceability and sustained performance. Since Flash modules do not have the same heat, vibration or harmonics issues that plague mechanical drives, more modules can be implemented in a smaller amount of space. As discussed earlier, this space utilization is also what plagues legacy arrays that use SSD in drive slots.


By focusing on the aggregation of Flash modules, exclusively systems like Violin Memory’s 3200 Memory Array is able to pack 10TB’s of SLC based Flash storage into a 3U appliance. This density becomes the cornerstone feature of the Memory Array because it not only drives down cost it significantly improves performance.


With the increase in Flash density there is a significant decrease in the cost to deliver capacity. To match the 10TB capacity of the 3U Memory Array a typical legacy storage system would have to use 7 to 10 storage shelves fully populated with SSD. That adds approximately $50,000 to the cost of the system just in shelves. More realistically the legacy storage shelf would not be able to support the performance capabilities of 15 or more SSDs in a single shelf, more likely it would take 2 to 3 times as many shelves, 1/2  to 1/4 populated, to deliver the expected performance which would result in adding another $50,000 to the cost of the system.


On a per Flash module basis Memory Arrays have a similar "erase then write" issue that SSDs do. The difference is that Memory Arrays have significantly more Flash modules per storage unit and these modules work together as a system to allow “Flash housekeeping” to be performed in parallel while under sustained data load. The Violin system for example was designed to provide sustained performance even when fully written by taking advantage of Violin’s Switched Memory architecture as well as their patent pending Flash RAID method of protecting data across the array at the same time guaranteeing any read will not be blocked by an erase, hence providing low “spike free” latency and sustained performance.


The Memory Array itself is also not encumbered with external performance bottlenecks that SSDs face when installed inside of legacy storage arrays. Memory Arrays often have the option of attaching to storage via multiple PCIe, Fibre Channel or 10GbE connections and do not require invasive access to the server. They are also not burdened by having to support a complete compliment of storage software functionality like snapshots, replication and thin provisioning. If this capability is needed it can be added via a storage virtualization appliance or the operating system / hypervisor, but the Memory Array’s primary focus is on sustained performance and “spike-free” latency.


Flash Memory Arrays may be the key to wide spread adoption of solid state storage in the enterprise. Their density, sustained performance and predictable latency provides key advantages in dealing with performance related problems commonly seen in the enterprise data center while also providing data protection, hot-swap serviceability and affordable cost per GB. In contrast it is difficult to scale SSDs in legacy arrays due to the cost issues faced when implementing a data protection strategy (RAID) which drive up the cost/GB while negatively impacting performance.


Storage Switzerland will address the data protection challenge and further detail the in next entries in the What is a Memory Array series.

George Crump, Senior Analyst

Violin Memory is a client of Storage Switzerland