Monday, October 11, 2010

Unfortunately the introduction of data protection solutions like RAID has only shined a bright light on the latency problem as it usually makes it worse. The SSD latency problem can be solved by using Memory Arrays with flash-optimized RAID algorithms instead of solid state disks for the active data set and then using legacy mechanical systems for in-active data. These devices allow solid state storage to deliver optimal total performance in the smallest amount of space, which further reduces the cost per IOP and makes solid state easier to justify in the enterprise.

Background On The Latency Problem

The Read-Erase-Write Bug

The first challenge in dealing with the solid state latency problem is that from a performance perspective a flash device’s best day is its first day. After the device is completely filled any flash blocks need to be erased before being re-used. Cleanup operations need to occur and these steps can cause a dramatic drop in performance on a per cell basis.

This performance drop is caused by how new data is written on a flash device once all the available blocks are written to. First, flash memory has to be written to pages, sequentially, within a block. Those blocks are not as granular as DRAM memory or even mechanical hard drives. A flash block is typically 128-256 Kbyte no matter what the size of the actual new or modified data is. Also these blocks, if they contain old data, need to be erased before they can be written to. The block to be reclaimed must have any active pages moved into other free blocks and then the retired block can be erased and reused.

These housekeeping operations occur in the same data path shared with the users, so they do impact performance. The time it takes to erase a block can be 2-10 milliseconds. With reads happening in 50 microseconds, they can suffer latency because of block reclamation activity if a read is attempted from a Flash die performing an erase. During the read-erase-write process all reads from that Flash die stop and a read queue begins to build. This mismatch of 50uS reads being held up by 10ms erases is what can cause very noticeable latency spikes in a busy enterprise environment.

The Read-Modify-Write Bug

This problem is compounded as data availability techniques are applied to flash technology, since the most common data availability techniques employ read-modify-write algorithms which combined with the read-erase-write requirement of flash described above deliver a double dip performance penalty. While RAID 0 (striping) and RAID 1 (mirroring) avoid the double dip problem they are insufficient for use in the enterprise. RAID 0 provides no redundancy and any failure of any drive in the RAID group will lead to data loss.

RAID 1 is considered an SSD best practice for integrators of the technology. These are most often used by legacy vendors with mechanically based storage systems. The problem with RAID 1 is that it requires a doubling of capacity, which is cost inefficient, especially in solid state storage. This cost inefficiency is what leads to the adoption of parity based data availability schemes like RAID 5 and RAID 6 in mechanical storage. If RAID 1 was considered too expensive for mechanical drives it is odd that it becomes the best practice for the more expensive SSDs.

RAID 1 earns the best practice moniker because RAID 5 and RAID 6, while more capacity efficient, are a latency and IOPS nightmare when applied to flash technology. This is because of the above mentioned read-modify-write nature of the protection schemes. More writes means more read-erase-write cycles which causes more queuing delays for reads. In fairness the RAID 5 and 6 algorithms can become more latency efficient or less latent as flash modules are added to the RAID group. This is because the read-erase cycle can be spread across more potential flash memory devices although it is impossible to predict when reads might contend with Flash die with blocks needing to be erased and reclaimed.

Working Around The Bugs

The traditional storage system manufacturers can’t realistically benefit from adding additional Flash memory to the storage system. Most traditional storage vendors implement flash technology as SSDs utilizing a Hard Disk Drive (HDD) form factor as well as the associated protocols, controllers, and management tools. Using the current HDD form factor takes advantage of the extensive disk aggregation interface already in place to mount and connect drives to the host system.

While using SSDs in traditional HDD shelves has provided a short cut to solid state storage for legacy vendors, their customers may not appreciate the path. The attempts at integration has quickly exposed the limitations of today’s aggregation technologies; a shelf that might normally house 24 HDDs may only support 2 SSDs before overtaxing the shelf’s aggregation interface. This means the rest of the shelf has to remain empty. To build a large SSD based RAID 5 group that would not see the performance impact from the parity overhead might take 10 or more shelves of SSD, each only a 1/3 or less full. For almost every environment this design would not be practical from a cost, power and space perspective.

The enterprise is then faced with a problem. There is a need for higher performance storage like solid state and recent years the individual component cost has become very attractive, but the cost to make solid state enterprise ready by adding data availability features like RAID make it cost prohibitive given the restrictions of the legacy storage systems. Reality is that this intersection of performance and cost can be met by breaking with legacy storage systems that were designed for another era and moving to a “solid state only” design like the Violin Memory Array.

Using The Memory Array To Fix The Latency Bug

The Density Bug Fix

The Memory Array takes advantage of a very dense population of memory modules or VIMMs (Violin Memory Modules) and a hardware based RAID innovation that is designed specifically for flash storage. It is specifically designed to fix the latency bug while providing a cost effective yet reliable alternative to traditional storage for storing the most active data set.

The Flash RAID Bug Fix

The density of the VIMMs is one of the key advantages to the Memory Array. The VIMMs are grouped into different RAID groups within the Memory Array. Each of these RAID groups consists of five VIMMs, 4 for data and 1 for parity. In the Violin case these VIMMs are hot swapable and actually have indicator lights that alert the user if it needs to be replaced. The Memory Array also has the capability to have modules fail in place and there are up to 4 active spares. In other words a failed module is logically replaced by another through the intelligence of the Memory Array. Essentially a solid state version of the global spare. As a result full protection levels are returned automatically without human intervention. The timing and replacement of modules is at the discretion of the user.

The Memory Array leverages these multiple RAID groups to be able to spread the protection workload out so that no single write or read causes a performance bottleneck. While data can be received by the Memory Array at sizes ranging from 512 bytes to 4 Mbytes, once in the unit it is segmented into 4k blocks and those blocks are then striped across the multiple RAID groups in parallel. This gives the user a protected data platform that maintains 80% of its capacity compared to the “best practice” RAID 1 algorithm that other SSD systems are forced to use. Efficiency of capacity is a critical component especially in the higher cost per GB world of solid state storage.

Fixing The Read-Erase-Write Bug While Maintaining Performance

However efficiency is useless if the performance, as a result of being space efficient, is negatively impacted on the solid state tier. The Memory Array architecture allows for the devices to read and write in parallel, not having to pause while an erase operation takes place because only 4 out of 5 VIMMs are needed to be read at any given moment. What this means is that even under heavy loads with RAID protection in place the Memory Array can sustain a constant level of performance and not have to deal with the severe latency spikes that SSDs do when used in legacy storage systems.

The Memory Array provides much greater resilience to component failures than traditional SSD drives that legacy manufacturers are using. Single chip failures do not cause data loss or modules to fail and be replaced. The systems leverage the capabilities of solid state technology, density, fast rebuilds, etc... Most importantly it significantly reduces the latency incurred when trying to apply traditional RAID protection to solid state based storage. As a result the Memory Array becomes the ideal solution for most if not all of the active data in the enterprise and can be counted on as a reliable enterprise grade storage device.

George Crump, Senior Analyst