Architecturally, the typical Exadata X2-8 configuration consists of two high powered database servers with plenty of I/O capability. Each system includes, among other things, a 512MB battery backed write cache, eight 300GB 10K SAS disk drives, eight InfiniBand QDR (40Gb/s) ports, eight 10Gb Ethernet ports and eight 1Gb Ethernet ports. The storage is typically 14 Exadata Storage Servers, each with twelve 600GB 15,000 RPM High Performance SAS disk or twelve 2TB 7,200 RPM SAS disks. Each Exadata Storage Server also includes four 96GB PCIe SSD cards. The 14 Storage Servers and the two database servers are connected via three 36-port QDR Infiniband switches.

This is a big system with a big price tag (over $9 million), and over $6 million of that cost is in software. Is it worth the investment? It may be if Exadata can solve problems that no one else can. There are specific Exadata features that can’t be had on a standard Oracle platform. The first of these is SmartScan which should reduce the data volume for specific types of searches, primarily non-indexed searches like full table scans. It doesn’t provide much benefit, however, on indexed based searches.

Exadata also includes FlashCache, a feature that takes advantage of the four SSD cards installed in each of the storage servers. This augments but doesn’t replace the standard FlashCache capability at the database level, and as a write-through cache, is only useful for read heavy environments. Also, it’s important to note that this is an external cache removed from the database application and is not as fast as a local in server database cache.

Not all workloads are composed of large block reads; random I/O is still heavily used in database accesses and will continue to be so. The addition of flash to the Exadata platform was meant to address random I/O, but it is undersized at 384 gigabytes per cell to cover anywhere from 2-5 usable terabytes of data per cell. This undersized flash cache is likely to result in many cache misses in a typical production environment. A cache miss actually hurts performance since extra time must be taken to look for the data in cache first before going to the mechanical hard drives.

Exadata includes Automatic Storage Management (ASM) – as does any standard Oracle environment – which helps Exadata storage server performance. It can subdivide disks, creating virtual disks or what Oracle calls “cell disks” from whole physical disk drives. The cell disks are then split into “hot” and “cold” areas. The outer edge of the platter is for hot data and the inner sections are used for cold data. This is essentially classic short stroking with the ability to use the inner part of the platter. This is not unique by today’s storage standards; many systems can intelligently place data on different sections of a disk based on access. However, ASM does not have knowledge of the flash that is embedded in the storage servers.

The Exadata Storage cell also provides the I/O Resource manager (IORM) which is separate from ASM and is only available on Exadata hardware. The IORM provides for I/O resource management at the intra-database level, controlling the distribution of I/O resources between multiple databases. This control of I/O resources is critical if multiple databases are storing data on each cell to make sure I/O blocking doesn’t occur. Of course if low latency, high throughput resources such as SSD technology replace the low-technology disk drives, the need for I/O resource management, while not completely eliminated, is greatly reduced.

Next, Exadata includes storage Indexes, which are created and maintained by the storage servers themselves. This feature was designed to compensate for the database being scattered across different storage servers. It allows Exadata to better determine which storage servers are most likely to have the data needed when responding to a WHERE clause. This will improve SQL operations and run dramatically faster because a high number of them are replaced by a few lookups. However if the data were all on one extremely fast storage system their need would be questionable. In addition, the ability of storage indexes to deal with changing data is questionable.

The final Exadata unique feature is called Hybrid Columnar Compression (HCC). It optimizes data storage requirements while avoiding some of the performance issues associated with compression. HCC requires data to be loaded using data warehouse bulk loading techniques. It can provide data optimization rates as much as 15X normal capacity requirements but may also cause a noticeable performance loss, especially with volatile data. Essentially, most recommendations are to use HCC on infrequently accessed OLTP data and to only set the compression level to “low” or 4X.

The hybrid columnar compression that is so effective for full table scans is another  determinate to the use of flash for random I/O. The data in the flash cache is kept in compressed form, so that if any row is needed, the entire 32 KB compression unit needs to be stored, even if only a fraction of the units’ data are actually needed. The compression increases the size of the cache needed for full scans, but it ends up

dramatically reducing the available size of the flash cache for random I/O. Updates to rows in the hybrid columnar compressed form are much more complex than a standard update since the full compression unit needs to be retrieved, uncompressed, modified, recompressed and restored on the drives.

Since there are only 12 disks in a storage server, the amount of time it takes to fill the cache for random I/O workloads is so long that the access pattern is likely to change prior to effective cache reuse. This is a worthwhile tradeoff in a system that has many more full scans than updates, but is not ideal for a system that is highly transactional

or has a mixed workload.

Exadata’s storage challenges are created because it takes a ‘sledgehammer approach’ to database performance problems. This brute force strategy for achieving a faster database experience is something that most data centers don’t need and probably cannot afford. Also all this investment in servers and storage can only be used to run one application – Oracle. If there is a decision to move to a different database platform or if another application in the environment would benefit from this level of performance it can’t be done with Exadata hardware.

The Exadata Alternative

If one truly needs to design a system similar to an Exadata solution and has that kind of budget it can certainly be done more cost effectively. One example is a pure solid state storage solution similar to the one that Oracle Guru Mike Ault described in a recent webinar for Texas Memory Systems. In this configuration multiple SSD appliances are used for the primary storage area of the database and PCIe SSD cards are used in the servers to act as local FlashCache.

The advantage of this solution is that there is no spinning disk at all, which means no need for storage indexes or ASM disk dividing. The entire disk is fast so you don’t need to separate it into hot and cold regions and, since it’s a single, highly responsive entity, there is no need for indexing sub-groups of storage. In addition, a local cache miss happens very quickly since the cache is in the application server, so the cache miss’ effect is negligible; there is also no chance of a cache miss in storage since the entire storage tier is solid state. This leads to simpler information and fewer parameters to be concerned with. The local use of PCIe SSDs makes FlashCache more effective since the cache is now on the same I/O bus as the actual application that’s processing the data.

Enhance What You Have

The biggest storage challenge to Exadata is that it ‘clear cuts’ and obsoletes the investment in the storage assets currently in the data center. The reality is that instead of throwing away what’s on the floor, most Oracle performance issues can be solved surgically by leveraging the existing storage infrastructure.

First, solid state storage solutions from companies like Texas Memory Systems are available with Fibre Channel, PCIe or InfiniBand attachments. This means they can be installed with little or no change to the existing environment. The last thing most data centers want is yet another storage infrastructure to manage. It makes more sense to upgrade just one section to 8Gb FC, for example, and then leave the rest of the environment at 4Gb.

Second, most Oracle implementations do not need an entire hardware rip and replace to improve performance. Moving some or even all of the database onto solid state storage may be all that’s needed. This allows the current server hardware and storage to continue to play a role, eliminating the need to rebuy those assets.

Finally, there may be more powerful or affordable servers from one of the traditional server manufacturers. Using an alternative solution provides the flexibility to choose the servers that make the most sense. It also provides the ability to run multiple application types or to switch database environments altogether. Lastly, when it’s time to upgrade one of the components of the system, just that component has to be replaced. To date, in previous Exadata generations, Oracle has not typically provided point upgrades, instead insisting that the whole system be replaced.


Oracle Exadata is an attempt to solve some of the biggest challenges that enterprise Oracle environments face. It does this by providing a unique set of software features to enhance relatively old hardware. This creates a new set of challenges in that customers could end up with a more complex solution that they’re intimidated to touch and are permanently locked into Oracle for all components of it.

While its results are impressive a simpler approach may be to use modern storage hardware like solid state disk to surgically fix the performance solutions. This could maintain the flexibility and customizability that have been the hallmark of Oracle environments for the past 20+ years.

Texas Memory Systems is a client of Storage Switzerland

George Crump, Senior Analyst

Enhancing Server and Desktop Virtualization with SSD Series
  Part I - Cost Justifying SSD
  Part II - Integrating SSD into a Virtual Server or DT Infrastructure

 Related Content
 Will MLC SSD Replace SLC?
 SSD, Heal Thyself!
 The Importance of SSD Architecture Design
 Using SSS with High Bandwidth Applications
 Solid State Storage for Bandwidth Applications
Screen Casts
 Access our SSD Screen Cast