Thanks to the introduction of enterprise quality NAND flash, solid state devices (SSD) have become an increasingly viable option for IT professionals looking to solve storage performance issues. Over the past few years this has lead to the introduction of many forms of flash based SSD. Recently a new form factor has been gaining in popularity; PCIe SSD. The temptation is to treat all of these devices the same but the reality is that they’re not the same and need be examined more carefully.

Why PCIe SSD For Servers?

Installing flash based storage inside the server that needs a performance boost is a popular strategy for IT Managers considering the deployment options. By being local to the server, the performance of solid state is not impacted by the potential bottlenecks of a shared storage system or that of the connecting network storage architecture.

After the decision to be installed inside the server is made, there are still numerous choices to consider. Should the server be deployed with SSD drives, SSD DIMMs or PCIe based flash? In general, drive form-factor SSDs are ideal for booting servers and for servicing medium performance needs. PCIe is typically reserved for higher performance requirements, such as when the flash memory is going to be used as an alternative to additional DRAM.

PCIe SSD does not require a drive slot, so it can be installed into a server that’s fully loaded with hard disk drives. If implemented correctly PCIe has the potential to be faster than fibre channel attached SSD. It can also enjoy a performance improvement by avoiding the traditional storage protocol stack and gaining direct access to DRAM and CPU resources via the PCIe bus.

What Types Of PCIe Are There?

There are two basic types of PCIe based SSD on the market today. The first is the native PCIe SSD. It offers the performance benefits described above and does not use the traditional storage protocol stack. These devices use flash memory modules (not drives) for the flash storage. This allows them to be more densely packed on the board which leads to greater capacity and board reliability.

An alternative to native PCIe are SATA/SAS PCIe SSDs. These are boards that essentially have an SSD drive mounted on them and are then attached to a SATA/SAS controller also on the board. While this does allow for a direct boot from the PCIe board it has the performance overhead of the SAS/SATA interface. These designs are not as densely packed with flash memory and need room for the RAID logic. As a result they may have less capacity per board than do the native PCIe SSD solutions.

SAS/SATA RAID PCIe are often RAID-only configurations, meaning that some of the flash capacity is allocated to redundancy. While there are many cases where RAID data protection is required, it’s not always the case. For example, if the PCIe SSD is being used as a paging area for virtual memory or as a read-only cache, the user may want the option of bypassing RAID data protection, and instead just count on the protection that ECC memory provides natively. Native PCIe SSD provides this flexibility of choice, without dedicating board real estate to RAID logic.

Differences Between Native PCIe SSD

Many data centers when looking to augment system RAM or develop high speed server based caches will lean toward native PCIe based SSDs. Even within these devices there are key differentiators that deserve consideration when determining which PCIe card makes the most sense for each specific use case.

The flash controller is the first area that deserves examination since it’s responsible for many functions on the SSD. The controller receives inbound data and makes sure that data is distributed evenly across all the available NAND flash, a process called “wear leveling”. It also handles functions like garbage collection, that pre-clears old flash cells to receive new data, so that a write does not need to be put on hold while the clearing process is run.

Onboard Or Off-board Controller Implementations

One consideration is understanding what provides the horsepower to these controller functions. Some suppliers put this processing power on the PCIe board itself, essentially dedicating a processor to the task. While this method may provide a more predictable performance model and requires no resources from the host, it’s harder to develop and raises the cost of the SSD.

Another challenge with using custom designed processors dedicated to the PCIe board is that the speed of the board is limited to the speed of the processors on it. It doesn’t benefit from an upgrade in server hardware.

Companies like Virident and others have developed an alternative option, to move the flash controller function to the processors of the server in which that PCIe SSD will be installed. Since these controller functions now become mostly software, solutions built on this strategy are easier to develop and are less costly to deliver. Also, as the host computers are upgraded or replaced with faster CPUs, this form of PCIe SSD will see performance improvements in the process.

The software controller design brings some concerns about its impact on the host CPU. The basic issue is that if the CPU gets busy writing and reading data to and from the flash this may negatively impact performance of other services running on that server.  While there could be a case made that this will cause some variability in performance, for most typical workloads the CPU has more than enough horsepower to manage these flash controller functions and still support the rest of the processes on the server.

Too some extent PCIe SSDs that use the onboard controller approach were not designed with the server-attach model in mind, where there will always be a processor available for them to use. Instead, they were designed to be placed inside a storage appliance that may not have as much processing power as a full fledged, server-class system. Of course, the reality today is that even storage systems have extensive processing power, making the off-board controller model acceptable in these use cases too.

The off-board approach can be extended even further. For example, Virident can create multiple virtual controllers to perform specific functions as demand warrants to handle peak loads. An onboard controller has a hard set processing capability and cannot typically scale or alter their function based on load.

A case can be made that the software based model will be the most viable going forward. To increase the efficiency of flash based technology the lithography of the NAND flash chips will continue to shrink. With each increase in chip density, the number of errors also increases. This means that the onboard controllers will need to increase in speed and size to keep up, something software controllers are ideally suited to address, as they simply leverage the ever increasing CPU capabilities of the host.

Host Based Drivers

The other consideration when selecting PCIe is that they require host oriented drivers in order to be recognized. While some may consider this a negative because a driver needs to be loaded on the host for the board to be recognized, it can actually become a distinct advantage since it allows the PCIe SSD to avoid the storage protocol stack.

Host oriented drives deliver better address translation. When a file system location is updated or changed it can be translated directly to a flash location in the host CPU environment. This means that the driver can perform this translation at the speed of a modern CPU (2.x to 3.x GHz), in the same way that virtual memory addressing is done. With other implementations address translation has to be done through SATA/SAS controllers after weaving their way through the storage protocol stack. This would be similar to offloading virtual memory addressing where measurable latency would occur.

Without direct address translation the writing or reading of data would have to ‘context switch’ multiple times from the application environment to the SATA/SAS RAID controller environment. This is error prone and latency bound when compared to the direct method employed by Virident and others.

Finally, the driver provides a better global view of all of the available flash on the board. With a SATA/SAS RAID approach the controller can only have access to the portion of the flash that has been provisioned directly to it. Think of it as five separate drives installed on the board. Each controller can only see one “drive” or in this case one flash area at a time. A direct driver approach can do wear leveling and garbage collection across all the flash in a single view.

Summary

PCIe SSDs are being chosen to solve server specific storage bottlenecks. Their performance potential is so great that they can be used as a nonvolatile and less expensive alternative to DRAM but this requires attention be placed on delivering extreme performance and removing latency. The native PCIe approach seems best suited for that and its off-board controller technology can provide an appealing alternative to hard wired controllers.

Virident is a client of Storage Switzerland 

Previous Entry: “How to Afford SSD For VDI or Virtual Servers”