How To Test SSDs
Challenges With SSD Performance Testing
As outlined in the previous article “How to Compare SSDs”, manufacturers’ product data can be used to make a ‘first cut’ when evaluating SSDs. But, the general lack of a standard testing methodology for the SSD industry means that data centers will need to test the devices in their own environments in order to make a sound final decision. With hard disk drive (HDD) technology the assumption could be made that the HDD itself was the I/O bottleneck. This meant any improvement in storage I/O capability would also improve application response time and that the application would always push the storage I/O to its maximum. With SSD technology, this isn’t always the case, as the hardware and software used to test HDDs often doesn’t push these solid state devices, which can have an order of magnitude greater performance. Consequently, greater scrutiny must be placed on the infrastructure used in testing SSDs.
Thursday, January 20, 2011
One of the biggest factors in measuring SSD performance is how the application is written and what the I/O capabilities are of the physical server in which the SSD is installed. Factors like server and disk I/O cache as well storage interface speed can also impact how extensively a test can measure the performance capabilities of the actual SSD. Even the operating system can impact how well the SSD performs.
Device Issues
While all of these factors need to be taken into account, most of them will be addressed as new versions of hardware, software and operating systems come out that are "SSD Aware”. However, the actual design and capabilities of the SSD itself can impact its reliability and performance now.
Steady State
The first recommendation is not to be too concerned about ‘fresh out of the box’ (FOB)performance. Most SSDs will perform significantly better initially and then after a period of time will show a performance drop off, which in some cases can be severe. So the first step in understanding which SSD is best for your environment is to make sure that the drive has been filled to capacity and has reached Steady State before doing any real testing of the device. Also, be sure to conduct this same ‘pre-conditioning’ routine on every drive that’s tested. As a point of reference, on the SNIA website, the Solid State Storage Initiative Technical Working Group has a Performance Testing Spec available that may be useful.
One Drive At A Time
The next step, after the drive has been inserted into the test environment, is to run the application set as close to ‘normal’ as can be simulated. It’s also important to avoid intermittent testing, and instead test one drive at a time, throughly. For example, don't test one drive on one day then another drive on another day; rather, try to test each drive for a week of continuous use. This provides a feel for how effectively the garbage collection process of the SSD controller will perform. After the first week of testing, perform a secure erase and run another week of testing. Obviously, measure performance throughout the process.
Test Caching As It Will Be Used
Most SSDs have a faster type of memory, often DRAM, that is used as a cache, similar to how it’s used with HDDs. This provides a more constant level of performance even in heavy write situations where program/erase cycles can become a problem or the garbage collection process needs to run in parallel with current write activity. It’s not uncommon for some SSDs to perform better than others, depending on the state of the cache setting. Also, cache sizes can vary between SSDs. For these reasons, cache should either be disabled before any tests are run or care must be taken to make sure the cache state that’s used is identical in testing as it will be in production.
Measuring Reliability
Most enterprise class SSDs will make it comfortably through even the most extensive two-week testing period, so measuring reliability may be difficult. Fortunately, there is a parameter that will give details on remaining SSD life and should be accessible for every SSD manufacturer - although getting to the information may be different for each drive. This parameter leverages the Self-Monitoring, Analysis and Reporting Technology (SMART) standard for drive diagnostics and provides a special SSD set of attributes that covers aspects like program (write) failures, erase failures and power loss. The most valuable for determining drive reliability is the parameter that reports on the program/erase cycles that remain. Think of this as the SSD equivalent to reporting how much ink remains in an inkjet printer. Comparing these two numbers after two weeks of testing described above should provide a reasonably accurate estimate of drive reliability.
Selecting the right SSD is a two-step process of using the reported performance statistic to develop a ‘short list’ of SSDs and then running the hands-on evaluation. As discussed in our prior article “How to Compare SSDs”, it is important to understand exactly what condition the SSD was in when the performance data was generated, to choose devices with comparable endurance ratings and to have a confidence level in the manufacturer. Then with the short list, create a test environment that mimics the production environment as closely as possible and work through testing, one drive at a time.
George Crump, Senior Analyst
Micron Technology is a client of Storage Switzerland
