NVMe over Fabrics (NVMe-oF)
NVMe performance over networking fabrics
NVMe-oF is an implementation of the NVMe standard that enables the use of the NVMe protocol across storage networking fabrics. With performance just shy of direct-attached storage (DAS), NVMe-oF is an increasingly popular implementation of NVMe that overcomes the challenges of relying on DAS for NVMe-level storage performance.
NVMe-oF has many benefits that make it an attractive choice:
- Storage disaggregation: by separating storage from compute, each can be scaled independently according to the network operator’s needs.
- Fabric agnostic: NVMe-oF can be used across virtually any networking fabric, e.g., Fibre Channel, RDMA (InfiniBand, RoCE, iWARP and more), TCP and other networking fabrics.
- Limited performance impact: NVMe-oF offers speeds near local NVMe with a close to negligible impact on storage latency.
Because of the inherent flexibility and performance of NVMe-oF, it can be implemented in a wide range of environments to unlock networked NVMe performance and limitless scalability.
Exascend offers its enterprise series NVMe SSDs with firmware enabled for NVMe-oF, making integration into next-generation storage infrastructure a breeze.
Storage access comparison
Storage access with NVMe-oF
With NVMe-oF, compute and storage are disaggregated. That means that a virtually unlimited number of computers can connect to pooled NVMe storage resources according to each client’s need.
Because of the disaggregation of compute and storage, each can be scaled independently without regard for the other. For instance, the NVMe storage pool can be expanded when storage requirements increase without scaling compute resources. Likewise, client devices can be upgraded, removed or change in numbers without requiring any changes on the storage side.
That allows greatly increased flexibility in scaling compute and storage resources – and in the case of NVMe-oF – without a significant tradeoff in storage performance.
Storage access with traditional, local NVMe
With a traditional NVMe storage setup, computers are equipped with local NVMe storage devices using the PCIe bus. While that allows fast direct-access to storage resources, compute and storage resources are tied together. For example, scaling up storage requires a free PCIe slot on the host device, otherwise the original NVMe storage device must be replaced.
Moreover, without pooled NVMe storage resources, computers need to utilize traditional networking protocols to access NVMe storage on other devices, an inherently limited way of managing shared storage.
Of course, other types of shared storage are more commonly used, each with performance far lower than local NVMe and pooled NVMe-oF storage resources.
NVMe-oF storage infrastructure
A better way to manage networked storage
Zooming in on the NVMe-oF storage infrastructure we can see that the clients or compute nodes connect directly to the NVMe storage pool through NVMe-oF using an adapter, for example a compatible InfiniBand or Ethernet adapter.
In essence, NVMe-oF acts as an abstraction layer between the compute and storage nodes, allowing NVMe commands and responses to be sent over a distance without involving the CPU for translation on either side, thus eliminating any unnecessary latency overhead.
On the host side, a storage controller transmits commands and responses directly to and from the PCIe-attached NVMe storage devices.
The result of all of the above is blazing-fast storage access over a network with a close-to-negligible increase in latency compared to local NVMe storage. The lion’s share of latency, just like with local NVMe, comes from the NAND flash on the NVMe SSD itself.
Types of fabrics used in NVMe-oF
Network fabrics for any occasion
NVMe-oF works flawlessly across a wide array of network fabrics, notably Fibre Channel, TCP and RDMA – including popular implementations such as InfiniBand, iWARP, RoCE and iSCSI Extensions for RDMA (iSER). That means that it can virtually plug straight into existing network or storage infrastructure without requiring the replacement of anything between the clients and the storage array. While each network fabrics has its pros and cons, NVMe-oF is network fabric-agnostic, allowing the network operator to unlock the power of NVMe in their preferred networking environment.
Minimal latency overhead
Remote storage, local latency
The NVMe-oF standard is designed to provide local NVMe-like performance across a distance, and latency is no exception. The specification allows a latency overhead compared to local NVMe of a minuscule 10 µs.
In practice, approximately 90% of the total latency is pure TLC NAND flash latency with the remainder made up by software latency. NAND flash controller latency is virtually nonexistent in NVMe SSDs – and the same rings true for NVMe-oF. For comparison, in traditional hard disk drives (HDDs) and SAS TLC NAND SSDs, the controllers alone contribute around 20–25 µs of latency.
What NVMe-oF’s minimal latency overhead means in practice is that the storage network provides instantaneous response times just like with local NVMe storage. With NVMe-oF storage access, gone are the days of slow networked storage that is a clear downgrade from DAS.
Bring NVMe-oF to life with Exascend's enterprise SSDs
Enterprise-class SSDs ready for NVMe-oF
Exascend is passionate about the future of networked NVMe storage with NVMe-oF. That is why we are working continuously to make the most out of the NVMe-oF standard with our flash storage solutions.
All of our enterprise-class NVMe SSDs, including form factors such as U.2, M.2, and E1.S, are fully compatible and optimized to bring out the best of the NVMe-oF specification. That means they deliver the local NVMe-level performance across a distance that the specification promises.
NVMe-oF SSDs customized and optimized to perfection
Even better, our commitment to the NVMe-oF standard also means that Exascend’s exclusive technologies and unique customization services are available in NVMe-oF-capable SSDs. For network infrastructure operators, this brings an incredible level of flexibility to deploy NVMe-oF in essentially any environment – including environments previously prohibitive to enterprise drives such as the rugged edge and applications where precise performance profiles must be achieved.
Our services and expertise
Engineered for NVMe-oF
Our NVMe-oF-capable enterprise SSDs are engineered to work flawlessly with the NVMe-oF standard. That is how we unleash the full power of NVMe-oF in your application.
Enterprise performance everywhere
Bringing enterprise-class storage performance to the most demanding environments is a key part of our company DNA. We want to take NVMe-oF where it has never been.
Customized like no other
We pride ourselves with offering the industry’s most extensive customization services. Whether you need some mild tuning or something completely new, we’re ready to deliver.
The technology leader
We are leading the push toward faster, more reliable and fundamentally more flexible flash storage solutions with unique technologies and uniquely capable storage products.
Our recommended NVMe-oF SSDs
PE4 series
The PE4 series is Exascend’s enterprise-grade lineup of high-performing PCIe 4.0 devices. With extreme sustained read and write speeds along with Exascend’s technologies, the PE4 series brings the PCIe 4.0 interface to the next level.
Interface
Flash
Design
Up to
Sustained read
Up to
Sustained write
PE3 series
The PE3 series brings high-level performance to enterprise applications without compromising on stability. Available in a wide variety of configurations and storage capacities, the PE3 series fits any demanding enterprise application.
Interface
Design
Up to
Storage capacity
Up to
Sustained read
Up to
Sustained write
