Author: Ron Herrmann, Director of Sales Engineering

E8 Storage recently published record breaking SPEC SFS™2014 performance results with IBM Spectrum Scale™ (aka GPFS), proving that our innovative software architecture delivers high performance and low latency.  This was a big milestone for E8 Storage and our partnership with IBM, and it allows us to provide Customers a solution to tackle high performance file based workloads.

What is IBM Spectrum Scale?
At its core, IBM Spectrum Scale is a distributed file system which is deployed as a software-defined storage solution.  It uses block storage devices (SAN or direct attached) for data storage and creates a file system with a single namespace distributed across many nodes in a cluster.

There are two ways that IBM Spectrum Scale attaches to storage:

  • Network Attached (e.g. SAN) – block storage is shared to all nodes in the GPFS cluster, typically via the Fibre Channel protocol. This mode is generally used with SAN arrays.
  • Direct Attached via Network Shared Disk (NSD) – the GPFS NSD servers provide access to storage that is visible only to those servers. This is generally used for SAS attached JBOD/JBOF storage, but supports local storage as well.

E8 Storage provides block storage protected with RAID 5/6, with the ability to create shared writable volumes required for GPFS clients.  That gives E8 Storage the ability to be a SAN replacement and be deployed in network attached mode (without NSDs), which is the configuration used for our SPEC SFS™2014_swbuild submission.

Why deploy with Network Shared Disk (NSD)?
The E8 Storage architecture has the flexibility to be deployed in both SAN mode and NSD mode, depending on the needs of your application environment.  While SAN mode is a simpler model to deploy, there are merits to deploying E8 Storage in NSD mode.

  • Expanded OS Support – E8 Agents are supported on Linux clients, but if your cluster includes other operating systems such as Windows, using NSD mode is the way to go.
  • Expanded NFS Client Support – Operating systems that are not supported natively by IBM Spectrum Scale (such as Mac) can attach to NSDs via NFS exports for access to shared storage.
  • Supporting clients without RDMA – If the clients are using embedded Ethernet (such as Intel LOM) or otherwise don’t have RoCE/IB support for communicating to E8 Storage, they can connect via their existing protocol to NSDs.


E8 Storage attached to Network Shared Disk (NSD)
E8 Storage attached to Network Shared Disk (NSD)

Accelerating IBM Spectrum Scale with Network Attached Mode
While NSD mode provides expanded support options, we recommend network attached mode to unlock the full performance of E8 Storage and simplify your IBM Spectrum Scale deployment.  Here’s a look at the benefits of network attached mode with E8 Storage.

  • Performance – E8 Agents on the GPFS clients communicate directly with the E8 target, delivering the best performance with the lowest latency. Addings NSDs to the data path adds performance overhead, especially with regards to latency.
  • Less Hardware – NSDs are additional servers that have to be added to the cluster specifically to process I/O requests. While 4 strong servers with 2-3 100GbE / IB ports can drive the performance of an E8-D24, it’s still a hardware footprint that can be avoided with direct mode.
  • Licensing – With IBM Spectrum Scale 4.x and earlier, each node / node role required additional licenses at additional cost, which are eliminated with direct mode. This is no longer true with the new IBM Spectrum Scale 5.0 Data Management Edition licensing model with capacity based licensing.  More information is below on licensing.


E8 Storage in Network attached mode over Ethernet / IB
E8 Storage in Network attached mode over Ethernet / IB

Now is a good time talk about the node roles in GPFS since our diagrams have shown a couple of different deployment options.  It’s important to note that whether a host is designated as a GPFS client or GPFS server, all nodes take an active part in the distributed file system cluster.

When a node is added to the cluster it can be designated as a client only, or perform additional server functions for the cluster.  Any node can take on any role, it’s simply a configuration option.  The server functions are:

  • NSD: providing access to block storage for all clients in the cluster, and responsible for block I/O operations
  • Cluster management services: cluster manager (at least 1), quorum managers (at least 2, but not more than 7)
  • Cluster export services: exporting an NFS / SMB server to non-GPFS clients (e.g. Mac)

Whether client applications are running on the same host as the GPFS servers is up to the IT manager and their needs.

Hybrid mode for maximum deployment flexibility
One of the benefits of the E8 Storage architecture is the ability to be deployed in both SAN and NSD mode at the same time.  It’s possible to install E8 Agents on 2-4 NSDs as well as connect other GPFS clients with E8 Agents directly to E8 Storage at the same time.  This caters to both approaches depending on the quantity and type of applications that the customer is running.

A short note on IBM Spectrum Scale licensing
Customers are asking us about the new IBM Spectrum Scale 5.0 licensing model, and while it’s not our software it’s worth a quick overview.  More information can be found in IBM Spectrum Scale 5.0 announcement under Product Planning, or contact your local IBM Reseller for more information.

The new licensing program is IBM Spectrum Scale Data Management Edition, and is sold per-terabyte rather than per node.  This new licensing model provides several benefits to customers:

  • All IBM Spectrum Scale features are unlocked, including encryption, compression and disaster recovery.
  • The per-terabyte license is all inclusive, and no longer requires socket or node counts (which are still part of Standard Edition licensing). When a customer wants to add more clients or NSDs, there are no additional fees.

The IBM Spectrum Scale Standard Edition licensing remains available, and is licensed per socket depending on the designation of the cluster node (server or client).  To know whether you require a client or server license, refer to IBM’s licensing requirements below.

IBM Reference on GPFS Clients Licensing Constraints:
Computers which are directly attached to storage and meet all of the following criteria only require an IBM Spectrum Scale client license:

Contact your IBM Spectrum Scale Seller or Reseller for specific licensing detail and pricing

E8 Storage is a proud partner of IBM and we’re working with channel partners in the US and EMEA to deliver integrated solutions for data hungry applications with IBM Spectrum Scale.  Contact us to learn more!