back to article The Fibre Channel NVMe cookbook: QED from a storage whizz's POV

OK, you think Fibre Channel-based NVMe fabric access is a good idea. How do you implement it? Where is the FC-NVMe cookbook? Consulting technologist Greg Scherer told us about NVMe over Fibre Channel and Ethernet in what has become the first interview of two. He said FC-NVMe was a practical proposition, and we asked him how it …

  1. Chris Mellor 1

    Mistaken conclusion

    A vendor contact sent me this:

    I think you might have made a mistake in your conclusion in this article.

    In the last paragraph you say:

    The destination storage array will provide data services by using "a global file system that handles replication/data protection at the file-system layer (HDFS, MongoDB, Casandra, NFS, etc.)". Comment from storage vendors will be most welcome. ®

    I think if you re-read Greg’s comment, he talks about running global file systems outside of an array as an alternative to putting data services inside an array behind an NVMe namespace. So, he is suggesting you will either do it one way or another – without an array and using distributed software on hosts (i.e. like Excelero, or the apps he lists above), or the services will be contained within an array, like what Pavilion (and the NVMe-over-FC products down the road) does.

    1. melbatal

      Regardless of the conclusion, the essence of the message still stands!

      No one can deny the fact that FC is still the most relevant SAN infrastructure in today’s enterprise data-centers. Also, no one can deny the market trends that are showing NVMe attached Solid-State Storage Devices are destined to take over the SSD market in volumes and revenue from both SAS and SATA by the start of the next decade. Therefore, enterprise storage vendors should embrace the NVMe-Over-FC(fabric) architecture and benefit from the combined NVMe storage cost and performance benefits, as well as the FC interface infrastructure manageability, maturity, stability, reliability and data-integrity. The combined solution will bring the best of both the new and the old to the enterprise consumers.

      It is a well-known fact that NVMe-Over-Fabric by design was never limited to Ethernet or IB fabrics. The standards enabled FC fabric interconnect, for its many enterprise advantages including Guaranteed-Delivery and native RDMA support. That said, FC is not as ubiquitous as Ethernet, and it is typically less latency optimized than IB. Also, the current FC IOC vender solutions are still using legacy IOC drivers to configure and handhold every IO driven through their chips, which is a fatal flaw in the architecture if one wants to scale to millions of IOPs.

      Ideally, the NVMe-Over-FC(fabric) silicon vendors should allow their solutions to evolve to working with standard NVMe-Target/Initiator ports in a “Direct” model without requiring a chip driver or consuming tones of CPU and Memory bandwidth from the AFA controllers, which would place them in the same league with the Ethernet and IB NVMeoF vendor solutions from an IOPs perspective.

      I personally would venture to say that it is to our advantage as a storage community to push NVMe-over-FC(fabric) ahead along with IB and Ethernet in our Seagate AFA product, since most of our customers already have a FC SAN of some sort in their datacenter and would benefit from such solutions, and we plan to partner up with the traditional FC silicon vendors to provide more advanced NVMe-Target/Initiator Direct solutions.

      Mohamad ElBatal

      Seagate - Enterprise Storage Systems CTO

  2. J_Metz

    Almost there

    Greg is a great guy and very knowledgable about storage technology. Unfortunately, there are still a few things that need some correction/clarification.

    'NVMe has no concept of controller and LUN; the two concepts are merged in an NVMe "Name Space".' I'm afraid this is simply incorrect. The NVMe Controller is a major part of the NVM Subsystem. It is the NVMe controller that not only creates and maintains the access to the NVMe Namespace (the equivalent of a LUN in NVMe-speak), but is also the structure in a NVMe solution that handles the Queue Pair (QP) relationships.

    Not only are NVMe Controllers not equivalent to Namespaces (or combined), it is possible to have "public" namespaces that are accessed by more than one NVMe Controller.

    From there, many of the optional features in NVMe (and NVMe-oF) are managed by several controller functions that are indigenous to the relationship between the host and the NVM Subsystem that are independent of the Namespace(s).

    "For an array controller, they will layer services behind each NVMe name space (software and hardware)". Given the previous reference to layering services "on top" by way of the storage controller (i.e., not NVMe Controller, but the array controller), this is also incorrect. The architecture for the NVM Subsystem does not place services in between a namespace and the corresponding media, (i.e., "behind the namespace.").

    It is true, however, that several companies (both startup and established) have created mechanisms for enhancing services both within and outside of the NVMe controller, but no "array functionality" is inserted between namespaces and media to my knowledge. (Note: this is not to say that someone couldn't create, or possibly doesn't create, a "meta-namespace" in which array functionality could be inserted in between, but this is not a standards-based approach).

    "For the array case, for all practical purposes they will advertise multiple Name Spaces, typically one per exported storage area (read this as Controller + LUN). This will be virtualised storage, so each exported storage area will be RAIDed, replicated or erasure-coded." This is mostly true, but with one minor tweak. Storage arrays that do this are likely to treat itself as both the host and the storage target.

    That is, the multiple namespaces that it manages will not be presented directly to the hosts. Instead, what happens is that the array controller will often establish individual namespaces with each NVMe drive, which in turn will be aggregated by the array controller (just like you would find with any other disk volume). Then, on the host-facing side, the array will maintain a separate target functionality and present a NVMe target to the corresponding host.

    Why is this important? Because the QP relationship between the actual host and the storage media is not contiguous. There are trade-offs - both positive and negative - to doing this, so it's neither a 'good thing' or a 'bad thing.' It's just an architectural implementation choice, that's all. But it is important when comparing to the "distributed file storage" mechanisms that are also discussed here.

    With respect to FC-NVMe, "Essentially the effort is to export a virtualised LUN in the form of an NVMe NameSpace, and register this name with the FC switch NameServer; so it can be discovered by FC-NVMe Initiators." This is 100% correct. :)

    I would also make a comment about the last paragraph ("Comment+"), but I see there is a clarification here in the comments already. :)

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019