Monday, January 10, 2011

What is PCIe SSD?

PCIe SSDs embed solid state storage on a PCIe card. That card is installed in a server and appears as another internal drive. Of course this drive, unlike a hard drive, is incredibly fast. By leveraging the PCIe bus inside of a server these cards are also less expensive per GB than their external solid state storage brethren, espcially when the dollar per IOP is considered. They utilize the power supply and casing of the server which results in lower costs since these cards don’t have to provide it themselves. The total on card maximum capacities of these PCIe based SSD range from about 500GB per card for SLC Flash based systems and about 2 TB for MLC based systems.

The PCIe SSD Advantage

For SSD PCIe has an inherent advantage over other connectivity options, with very low latency and extremely high bandwidth. PCIe SSD has a direct channel to the server’s processor and as a result data can be delivered quickly. Compared to other connectivity options like fibre channel, Ethernet or even Inifiniband, this gives PCIe SSD cards a significant advantage. They are ideal for solving server-specific application response time problems caused by underperforming or overworked storage.

The PCIe SSD Disadvantage

What has held PCIe SSD back from broad adoption is the very PCIe connectivity that gives it such stellar performance. Sharing a PCIe card that is embedded in a single server can be difficult and is a challenge that many PCIe SSD vendors have wrestled with. A second challenge is one of availability and redundancy. While two PCI SSD cards could be mirrored internally, how does the storage manager provide protection if there is a failure of the entire server? A final challenge is overall capacity or system density as most servers have a finite number of PCIe slots which are often needed for other purposes. This leaves only one or two slots available for PCIe SSD cards, limiting the effective capacity to 1 or 2TB per server, which may not be sufficient for the given application. Ironically the opposite is also true - some servers need less than a single drive’s capacity but still want max performance. In short the inability to provision PCIe capacity up and down and use the total capacity more effectively is a key shortcoming.

The industry has begun to address these basic issues by forming the PCIe SSD Form Factor working Group. This group has taken up the challenge of incorporating the PCIe interface into the hot-swappable SAS/SATA 2.5” drive form factor. This common package provides a building block that can be used to solve the sharing and capacity problems with an array.

Sharing PCIe SSD

The performance advantage and potential cost advantage have led to customer demand for a sharable PCIe SSD solution. As a result each PCIe SSD vendor as well as server vendors and storage integrators have all attempted to come up with their own solutions.

A common use of shared PCIe SSD is to install it into a NAS appliance and to use the SSD’s performance in a caching tier to accelerate storage IOPS or as a separate stand alone tier of storage to hold specific sets of data. Several off-the-shelf NAS operating systems have the ability to run on virtually any server hardware and support a mix of storage vendors. Using NAS also comes with the overhead of IP and a file protocol like CIFS or NFS. This means that the speed advantages of PCIe SSD are only available to internal operations of the NAS and are lost once external communication with a server is initiated. This type of solution also may not address the capacity constraints of PCIe SSD. Most NAS devices have less available PCIe slots than a standard server since they often have need for higher than average storage and network connectivity. The other obvious challenge is that this is a file protocol solution and would not support block I/O (SAN) communication. Many environments that can justify SSD are going to be block I/O based and moving them to a NAS may be impractical.

To address the need for block storage some of the PCIe vendors have tried to develop their own storage chassis to allow for multiple PCIe SSDs to be shared across servers. Others have attempted to attach themselves to storage virtualization appliances and provide accelerated performance as a separate tier of storage. Once again these options involve the performance loss due to the overhead of the storage connection protocol, be it iSCSI or Fibre. There is also the overhead of the storage virtualization solution even if the connection between the virtualization appliance and the PCIe SSD hosting is PCIe based. Features like thin provisioning, snapshots and other data services common to storage virtualization all take a performance toll. While not noticeable given the latencies present in mechanical drive arrays this overhead would be a real bottleneck in the zero-latency SSD environment. Some SSD vendors have indicated a 30% loss in performance in this configuration. It is critical that to provide these features without a performance penalty by separating the data path from the control path.

Sharable PCIe Through I/O Virtualization

All is not lost in the attempt to share PCIe based SSD. I/O virtualization (IOV) technologies may offer the best solution to date. They build a network out of PCIe allowing multiple cards to be installed into a PCIe-only gateway which is then available to the connected hosts. An I/O gateway that supports off-the-shelf PCIe cards will have a distinct advantage in the near term, and the new 2.5” drives will eventually provide a common form factor.

As we discussed in our recent article “Offloading I/O from the Hypervisor with SR-IOV”, today, PCIe ‘sharing’ will likely be a 1:1 relationship between the PCIe card and server until support for SRIOV begins to appear next year. Even today though some PCIe SSD vendors are providing some form of sharing. Even without onboard sharing the 1:1 relationship provides the ability to dynamically ‘move‘ PCIe SSD still bringing flexibility. It allows PCIe SSD to be assigned and re-assigned as the need arises. For example if a financial system sees a spike in storage performance demand only during end-of-month processing then the PCIe SSD could be allocated to that server for that time period via software. Then, when the period is over it could be reassigned to another server as needed. In addition shared PCIe via IOV avoids both the redundancy and scalability issues that stand alone PCIe SSD and the other sharing solutions can bring about. For example a server failure is recoverable by reassigning the drive to another server.

The gateway allows for a large number of PCIe SSD to be installed in it or it can be used in conjunction with other cards. Also the gateways have complete redundancy built in or two gateways can be used with connectivity per host going between them.

How to Connect

There are three common connection methodologies available to connect server hosts to the I/O gateway: PCIe, Inifinband and PCIe over Ethernet (PCIeoE). While we detail these options in our article “Comparing I/O Virtualization Technologies” shared PCIe SSD further enhances the justification of PCIeoE. PCIeoE, as it is delivered by companies like Aprius, does not require a new and separate connection topology to be integrated into the data center, one that would require a new level of training and management. PCIeoE leverages the existing Ethernet cabling to provide simple connectivity, with very high scalability. It’s also not encumbered by the overhead of other IP based technologies like NFS, CIFS or iSCSI. This means that the PCIeoE connection delivers much closer to native (in-server) maximum performance. The PCIeoE card that is installed in the connecting server actually has dual 10Gb ports for up to 20Gb/sec of connectivity to the card, moving to 40Gb/sec in the future.

In addition to solving the shared PCIe SSD requirement for data centers, sharing PCIe SSD with PCIe over Ethernet based I/O virtualization technology represents an incredible opportunity for PCIe SSD manufacturers. It allows them to break out of the mold of just supplying drives and move up the food chain to providing more complete solutions.

George Crump, Senior Analyst