NVMe over Fabrics (NVMe-oF)

NVMe-oF servers in a data center

NVMe performance over networking fabrics

NVMe-oF is an implementation of the NVMe standard that enables the use of the NVMe protocol across storage networking fabrics. With performance just shy of direct-attached storage (DAS), NVMe-oF is an increasingly popular implementation of NVMe that overcomes the challenges of relying on DAS for NVMe-level storage performance.

NVMe-oF has many benefits that make it an attractive choice:

  • Storage disaggregation: by separating storage from compute, each can be scaled independently according to the network operator’s needs.
  • Fabric agnostic: NVMe-oF can be used across virtually any networking fabric, e.g., Fibre Channel, RDMA (InfiniBand, RoCE, iWARP and more), TCP and other networking fabrics.
  • Limited performance impact: NVMe-oF offers speeds near local NVMe with a close to negligible impact on storage latency.

Because of the inherent flexibility and performance of NVMe-oF, it can be implemented in a wide range of environments to unlock networked NVMe performance and limitless scalability.

Exascend offers its enterprise series NVMe SSDs with firmware enabled for NVMe-oF, making integration into next-generation storage infrastructure a breeze.

Storage access comparison

Storage access with NVMe-oF illustrated with a diagram.

Storage access with NVMe-oF

With NVMe-oF, compute and storage are disaggregated. That means that a virtually unlimited number of computers can connect to pooled NVMe storage resources according to each client’s need.

Because of the disaggregation of compute and storage, each can be scaled independently without regard for the other. For instance, the NVMe storage pool can be expanded when storage requirements increase without scaling compute resources. Likewise, client devices can be upgraded, removed or change in numbers without requiring any changes on the storage side.

That allows greatly increased flexibility in scaling compute and storage resources – and in the case of NVMe-oF – without a significant tradeoff in storage performance.

Storage access with traditional, local NVMe

With a traditional NVMe storage setup, computers are equipped with local NVMe storage devices using the PCIe bus. While that allows fast direct-access to storage resources, compute and storage resources are tied together. For example, scaling up storage requires a free PCIe slot on the host device, otherwise the original NVMe storage device must be replaced.

Moreover, without pooled NVMe storage resources, computers need to utilize traditional networking protocols to access NVMe storage on other devices, an inherently limited way of managing shared storage. 

Of course, other types of shared storage are more commonly used, each with performance far lower than local NVMe and pooled NVMe-oF storage resources.

Diagram showing storage access with traditional local NVMe storage.

NVMe-oF storage infrastructure

Diagram describing the overall architecture of NVMe-oF used in a storage network.

A better way to manage networked storage

Zooming in on the NVMe-oF storage infrastructure we can see that the clients or compute nodes connect directly to the NVMe storage pool through NVMe-oF using an adapter, for example a compatible InfiniBand or Ethernet adapter. 

In essence, NVMe-oF acts as an abstraction layer between the compute and storage nodes, allowing NVMe commands and responses to be sent over a distance without involving the CPU for translation on either side, thus eliminating any unnecessary latency overhead. 

On the host side, a storage controller transmits commands and responses directly to and from the PCIe-attached NVMe storage devices. 

The result of all of the above is blazing-fast storage access over a network with a close-to-negligible increase in latency compared to local NVMe storage. The lion’s share of latency, just like with local NVMe, comes from the NAND flash on the NVMe SSD itself.

Types of fabrics used in NVMe-oF

Diagram showing a selection of network fabrics that can be used with NVMe-oF.

Network fabrics for any occasion

NVMe-oF works flawlessly across a wide array of network fabrics, notably Fibre Channel, TCP and RDMA – including popular implementations such as InfiniBand, iWARP, RoCE and  iSCSI Extensions for RDMA (iSER). That means that it can virtually plug straight into existing network or storage infrastructure without requiring the replacement of anything between the clients and the storage array. While each network fabrics has its pros and cons, NVMe-oF is network fabric-agnostic, allowing the network operator to unlock the power of NVMe in their preferred networking environment.

Minimal latency overhead

Bar chart displaying the difference in latency between different types of storage.
Source: NVM Express®, Intel Open Source Technology Center

Remote storage, local latency

The NVMe-oF standard is designed to provide local NVMe-like performance across a distance, and latency is no exception. The specification allows a latency overhead compared to local NVMe of a minuscule 10 µs. 

In practice, approximately 90% of the total latency is pure TLC NAND flash latency with the remainder made up by software latency. NAND flash controller latency is virtually nonexistent in NVMe SSDs – and the same rings true for NVMe-oF. For comparison, in traditional hard disk drives (HDDs) and SAS TLC NAND SSDs, the controllers alone contribute around 20–25 µs of latency. 

What NVMe-oF’s minimal latency overhead means in practice is that the storage network provides instantaneous response times just like with local NVMe storage. With NVMe-oF storage access, gone are the days of slow networked storage that is a clear downgrade from DAS. 

Want to learn more about what NVMe-oF can do for you?

Bring NVMe-oF to life with Exascend's enterprise SSDs

Enterprise-class SSDs ready for NVMe-oF

Exascend is passionate about the future of networked NVMe storage with NVMe-oF. That is why we are working continuously to make the most out of the NVMe-oF standard with our flash storage solutions. 

All of our enterprise-class NVMe SSDs, including form factors such as U.2, M.2, and E1.S, are fully compatible and optimized to bring out the best of the NVMe-oF specification. That means they deliver the local NVMe-level performance across a distance that the specification promises. 

NVMe-oF SSDs customized and optimized to perfection

Even better, our commitment to the NVMe-oF standard also means that Exascend’s exclusive technologies and unique customization services are available in NVMe-oF-capable SSDs. For network infrastructure operators, this brings an incredible level of flexibility to deploy NVMe-oF in essentially any environment – including environments previously prohibitive to enterprise drives such as the rugged edge and applications where precise performance profiles must be achieved.

Our services and expertise

Engineered for NVMe-oF

Our NVMe-oF-capable enterprise SSDs are engineered to work flawlessly with the NVMe-oF standard. That is how we unleash the full power of NVMe-oF in your application.

Icon illustrating a connected cloud representing edge computing

Enterprise performance everywhere

Bringing enterprise-class storage performance to the most demanding environments is a key part of our company DNA. We want to take NVMe-oF where it has never been.

Icon illustrating Exascend's customization services

Customized like no other

We pride ourselves with offering the industry’s most extensive customization services. Whether you need some mild tuning or something completely new, we’re ready to deliver.

Icon with a lightbulb representing Exascend's technology leadership

The technology leader

We are leading the push toward faster, more reliable and fundamentally more flexible flash storage solutions with unique technologies and uniquely capable storage products.

Our recommended NVMe-oF SSDs

PE4 series

The PE4 series is Exascend’s enterprise-grade lineup of high-performing PCIe 4.0 devices. With extreme sustained read and write speeds along with Exascend’s technologies, the PE4 series brings the PCIe 4.0 interface to the next level.

PCIe 4.0

Interface

3D TLC

Flash

Enterprise-grade

Design

Up to

3,500MB/s

Sustained read

Up to

2,200MB/s

Sustained write

PE3 series

The PE3 series brings high-level performance to enterprise applications without compromising on stability. Available in a wide variety of configurations and storage capacities, the PE3 series fits any demanding enterprise application.

Photo displaying Exascend's PE3 series NVMe SSDs in the U.2 and M.2 form factors
PCIe 3.0

Interface

Enterprise-grade

Design

Up to

7,680GB

Storage capacity

Up to

3,100MB/s

Sustained read

Up to

2,000MB/s

Sustained write

Or are you looking for a customized NVMe-oF solution?
Scroll to Top

There's no time like the present.

Send us an inquiry