The server uses memory to store information that it will need immediate access to. It uses storage, on the other hand, to store information that it’s currently acting on, but doesn’t need at that specific moment in time. Memory, typically thought of as DRAM is known for high performance and low capacity - and for being expensive. Storage, typically thought of as hard disk drives (HDD) is known for high capacity, affordability but also know for much lower performance than DRAM. “Storage class memory” is a form of memory that has capacity and economics that are similar to storage but with performance that’s similar to memory.

As described above memory is the place where the processor stores information that it will need immediately. Ideally there would be enough of this memory to store all the data  that will be needed, but that unfortunately is not reality. Again, often thought of as DRAM, memory has challenges other than just cost. In the typical server there’s only space to hold enough memory to store a fraction of the total data set that the CPU needs to operate on. In addition to being expensive DRAM memory is also volatile, meaning that if a server crashes or loses power, information stored in memory is lost. Because of these factors most information is kept on storage for the majority of its lifespan.

Storage class memory

Storage is often thought of as a mechanical HDD that offers near limitless capacity, when compared to DRAM. It’s also persistent, meaning that information is not lost if the server crashes or loses power. The problem with hard drives is that in many cases they are unable to provide information to the application quickly enough.

Storage class memory is a new form of storage created out of flash-based NAND that can provide an intermediate step between high-performance DRAM and cost-effective HDDs. It can provide read performance similar to DRAM and write performance that is significantly faster than HDD technology.

Can all flash storage be storage class memory?

While many flash-based storage devices can offer compelling economics versus DRAM, in order to be considered an acceptable alternative to DRAM memory it must provide near-RAM performance. Simply putting any SSD into a server may not accomplish this goal. Certainly for most servers an upgrade to SSD in almost any form will provide some incremental performance improvement. But again this is when compared to HDD performance, not RAM performance.

Most flash-based devices will connect to a server via a traditional storage protocol (SATA or SAS for example). Unfortunately, while this connection provides for easy installation it can also become the biggest bottleneck to maximizing performance.

When looking for a flash-based device that will provide memory class performance, in other words storage class memory, the most ideal connectivity will be PCIe. This puts the flash-based storage in direct connection with the CPU providing similar bandwidth to that of DRAM memory, but without the need for legacy protocol translation.

Can all PCIe flash be storage class memory?

While all PCIe based SSDs share a common connection to the CPU they’re not all the same and therefore do not all qualify to be considered storage class memory. For example several cards on the market, although mounted on PCIe boards, are really just traditional SSDs connecting via SAS on the single board. This SAS connection becomes a similar bottleneck to performance as it would if they were using a SAS connection on the motherboard of the server.

Another key differentiator is the controller-to-flash module ratio. Some vendors in an effort to increase performance have added multiple controllers to their cards, each with its own designated area of flash. While the idea of more controllers may seem like a good one, in reality this design could be problematic. For example, each individual controller does not have access to all the available flash storage capacity and as a result performance can suffer, because the controller can’t stripe large data reads or writes across all the available flash cells.

Performance concerns

Another performance problem can be seen during garbage collection, the process of preparing old flash cells to receive new data. While each controller can do its own garbage collection it can only operate on the flash that it’s assigned to. This means even though there may be plenty of unused flash cells elsewhere on the board, they’re inaccessible to the other controllers. The result is these controllers will be forced to clear space for new writes, consuming finite ‘program/erase’ cycles. If this garbage collection  occurs during inbound write cycles the added overhead could significantly reduce write performance.

Also, because each controller only has at its disposal the flash it’s been assigned, other flash on the board isn’t available to replace the bad cells that accumulate over time. Eventually this may leave one controller with an inadequate amount of flash memory while other controllers have plenty of cells.

Finally, the multi-controller approach may not provide a performance boost to all types of applications. In most cases, getting high-performance from multiple controllers requires a multi-threaded application. Or the board needs to be used in environments where multiple applications can generate simultaneous single threaded requests. While certainly these situations exist they’re not necessarily the most common.

A storage class memory device made out of flash will need to perform at the full capabilities of the flash module in both multi-threaded application environments and single threaded application environments. Companies like Fusion-io are designing PCIe based solid-state storage that is memory class. By leveraging a single controller that has access to all the available flash modules it can provide high-performance in all situations.

Software versus hardware

The flash management intelligence and how that intelligence is delivered is another important consideration when examining storage class memory. The flash PCIe board can be designed to leverage the CPU and memory of the host, or it can have its own micro-controllers and embedded processors that handle the intelligence.

Again, on the surface it would seem that dedicated hardware on the PCIe board, like additional controllers and embedded processors, would be the ideal method. The challenge these multi-chip solutions is that they add complexity to the PCIe board and makes changes to the flash software more difficult.

While the complexity problem is somewhat obvious the ability to adapt to change is not. First of all, software engineers are constantly making improvements to any software they write, flash management software included. If specific silicon was dedicated to specific devices created, in most cases the vendor would need to deliver (i.e. sell) all that particular hardware prior to updating to a new software release. This delays the release cycle of new features and may even put the customer at risk, since bug fixes may be delayed as well.

The other advantage of using flash management software that leverages the server CPU and memory is that as new generations of CPUs and faster memory come out the flash device will get faster with each iteration. Fusion-io as an example, has cases where customers upgraded servers with newer, faster CPUs and instantly saw better storage I/O performance. With hardware-based flash memory management this would likely not have been the case.

Summary

Storage class memory fills an important gap between DRAM memory and HDD storage. It provides near DRAM performance at significantly lower costs and significantly higher capacities. However, when flash memory is positioned as a viable alternative to DRAM the expectations of that device are higher. It cannot be bottlenecked by legacy connectivity or require unusually high levels of application threading. Finally, it should be built around a flexible flash management architecture that allows for rapid design and the ability to leverage the investment that vendors like Intel have made in the advancement of CPUs.

Fusion-io is a client of Storage Switzerland

Previous Entry: “What Your SAN Fabric Manager Software Isn’t Telling You”