It's well known that data is often considered less secure while in motion, particularly across public networks, and attackers are finding increasingly innovative ways to snoop on and compromise data in flight. But risks can be mitigated with foresight and planning. So how do you adequately protect data in transit? It’s the next topic the SNIA Networking Storage Forum (NSF) will tackle as part of our Storage Networking Security Webcast Series.  Join us October 28, 2020 for our live webcast Securing Data in Transit.

In this webcast, we'll cover what the threats are to your data as it's transmitted, how attackers can interfere with data along its journey, and methods of putting effective protection measures in place for data in transit. We’ll discuss: 

• The large attack surface that data in motion provides, and an overview of the current threat landscape
• What transport layer security protocols (SSL, TLS, etc.) are best for protecting data in transit?
• Different encryption technologies and their role in protecting data in transit
• A look at Fibre Channel security
• Current best practice deployments; what do they look like?

Register today and join us on a journey to provide safe passage for your data.

It’s well known that data is often considered less secure while in motion, particularly across public networks, and attackers are finding increasingly innovative ways to snoop on and compromise data in flight. But risks can be mitigated with foresight and planning. So how do you adequately protect data in transit? It’s the next topic the SNIA Networking Storage Forum (NSF) will tackle as part of our Storage Networking Security Webcast Series.  Join us October 28, 2020 for our live webcast Securing Data in Transit. In this webcast, we’ll cover what the threats are to your data as it’s transmitted, how attackers can interfere with data along its journey, and methods of putting effective protection measures in place for data in transit. We’ll discuss:

• The large attack surface that data in motion provides, and an overview of the current threat landscape
• What transport layer security protocols (SSL, TLS, etc.) are best for protecting data in transit?
• Different encryption technologies and their role in protecting data in transit
• A look at Fibre Channel security
• Current best practice deployments; what do they look like?

Register today and join us on a journey to provide safe passage for your data.

As explained in our webcast on Data Reduction, “Everything You Wanted to Know About Storage But Were Too Proud to Ask: Data Reduction,” organizations inevitably store many copies of the same data. Intentionally or inadvertently, users and applications copy and store the same files over and over; with developers, testers and analysts keeping many more copies. And backup programs copy the same or only slightly modified files daily, often to multiple locations and storage devices.  It’s not unusual to end up with some data replicated thousands of times, enough to drive storage administrators and managers of IT budgets crazy. 

So how do we stop the duplication madness? Join us on November 10, 2020 for a live SNIA Networking Storage Forum (NSF) webcast, “Not Again! Data Deduplication for Storage Systems”  where our SNIA experts will discuss how to reduce the number of copies of data that get stored, mirrored, and backed up.

Attend this sanity-saving webcast to learn more about: 

• Eliminating duplicates at the desktop, server, storage or backup device
• Dedupe technology, including local vs global deduplication
• Avoiding or reducing making copies of data (non-duplication)
• Block-level vs. file- or object-level deduplication
• In-line vs. post-process deduplication
• More efficient backup techniques

Register today (but only once please) for this webcast so you can start saving space and end the extra data replication.

As explained in our webcast on Data Reduction, “Everything You Wanted to Know About Storage But Were Too Proud to Ask: Data Reduction,” organizations inevitably store many copies of the same data. Intentionally or inadvertently, users and applications copy and store the same files over and over; with developers, testers and analysts keeping many more copies. And backup programs copy the same or only slightly modified files daily, often to multiple locations and storage devices.  It’s not unusual to end up with some data replicated thousands of times, enough to drive storage administrators and managers of IT budgets crazy. So how do we stop the duplication madness? Join us on November 10, 2020 for a live SNIA Networking Storage Forum (NSF) webcast, “Not Again! Data Deduplication for Storage Systems”  where our SNIA experts will discuss how to reduce the number of copies of data that get stored, mirrored, and backed up. Attend this sanity-saving webcast to learn more about:

• Eliminating duplicates at the desktop, server, storage or backup device
• Dedupe technology, including local vs global deduplication
• Avoiding or reducing making copies of data (non-duplication)
• Block-level vs. file- or object-level deduplication
• In-line vs. post-process deduplication
• More efficient backup techniques

Register today (but only once please) for this webcast so you can start saving space and end the extra data replication.

The SNIA Cloud Storage Technologies Initiative (CSTI) recently hosted a conversation with Glyn Bowden from HPE that I moderated on “Using Data Literacy to Drive Insight.”  In a wide-ranging conversation just over 45 minutes, we had a great discussion on a variety of topics related to ensuring the accuracy of data in order to draw the right conclusions using current examples of data from the COVID-19 pandemic as well as law enforcement. In the process of the dialog, some questions and comments arose, and we’re collecting them in this blog. Q. So who really needs Data Literacy skills? A: Really, everyone does.  We all make decisions in our daily life, and it helps to understand the provenance of the information being presented.  It’s also important to find ways to the source material for the data when necessary in order to make the best decisions. Everyone can benefit from knowing more about data.  We all need to interpret the information offered to us by people, press, journals, educators, colleagues, friends. Q. What’s an example of “everyone” who needs data literacy? A. I offered an example of my work as a board member in my local police and fire district, where I took on the task of statistical analysis of federal, state, and local COVID-19 data in order to estimate cases in the district that would affect the policies and procedures of the service district personnel. Glyn also offered simple examples of the differences between sheer numbers compared to percentages, and how they should be compared and contrasted. We cited some of the regional variations of COVID data given the methodologies of the people reporting it. There are many other examples of literacy that were shared in the material, including some wonderful data around emergency service call personnel, weather, pubs, paydays, and lunar cycles. Why haven’t you started watching it yet? Remember its on-demand along with the presentation slides. Q. What’s the impact of bias in the “data chain”? A. Bias can come from anywhere.  Even the more “pure” providers of source data (in this case, doctors or hospital data scientists) can “pollute” the data. What you need to do to qualify the report is to determine how much trust you have in the provider of the data. Glyn cited several examples of how the filter of the interpreter can provide bias that must be understood by a viewer of the data. “Reality is an amplifier of bias,” was the non-startling conclusion.  Glyn made an interesting comment on bias: When you see the summary, the first questions you should ask are what’s been left out and why was it left out? What’s left out is usually what creates the bias. It’s also useful to look for any data that supports a counter-opinion, which might lead you to additional source material. Q. On the concept of data modeling.  At some point, you create a predictive model.  First, how useful is it to review that model?  And what does an incorrect model mean? A. You MUST review a model, you can’t assume that it will always be true, since you’re acting with the data you have, and more will always come in to affect the model. You need to review it, and you should pick a regular cadence. If you see something that is wrong in the model, it could mean that you have incomplete data or have injected bias. Glyn offered a great example of empty or full trash containers. Q. So, the validity of the data model itself is actually data that you need to adjust your assumptions? A. Absolutely. More data of any kind should affect the development of the next model. Everything needs to be challenged. Q. Would raw data therefore be the best data? A. Raw data could have gaps that haven’t been filled yet, or it might have sensor error of some type. There’s a necessity to clean data, though be aware that cleaning of the raw data has a potential to inject bias. It takes judgment and model creation to validate your methods for cleaning data. Q. Would it be worthwhile to run the models on both cleaned and raw data to see if the model holds up in a similar way? A. Yes, and this is the way that many artificial intelligence systems are trained. Q. Another question that could occur would be data flow compared to data itself. Is the flow of the data something that can be insightful? A. Yes. The flow of data, and the iteration of the data through its lifecycle can affect the accuracy. You won’t really know how it’s skewed until you look at the model, but make a determination and test that in order to see. Q. How does this affect data and data storage? A. As more data is collected and analyzed, we’ll start to see different patterns emerge in our use of storage. So, analysis of your storage needs is another data model for you to consider! Please feel free to view and comment, and we’d be happy to hear about future webcasts that would interest you.

There’s a fair amount of confusion when it comes to data reduction terminology and techniques. That’s why the SNIA Networking Storage Forum (NSF) hosted a live webcast, “Everything You Wanted to Know About Storage But Were Too Proud to Ask: Data Reduction.”  It was a 101-level lesson on the fundamentals of data reduction, which can be performed in different places and at different stages of the data lifecycle. The goal was to clear up confusion around different data reduction and data compression techniques and set the stage for deeper dive webcasts on this topic (see the end of this blog for info on those). As promised during the webcast, here are answers to the questions we didn’t have time to address during the live event. Q. Does block level compression have any direct advantage over file level compression? A. One significant advantage is not requiring the entire thing, the file or database or whatever we’re storing, to be compressed and decompressed as a unit. That would almost certainly increase read latency, and for large files, require quite a bit of caching. In the case of blocks, a single block can be the compression unit, even if it’s part of a file, database or other larger data structure. Compressing a block is much faster and computationally less intensive, which is reflected in reduced latency overhead and cache impacts. Q. You made it sound like thin provisioning had no overhead but on-demand allocation is an overhead and can be quite bad at the worst time.  Do you agree? A. Finding free space when the system is at capacity may be an issue, and this may indeed cause significant slowdowns. This is an undesirable situation, and the advice is never to run so close to the capacity wire that thin provisioning impacts performance or jeopardizes successfully writing the data. In a system with adequate amounts of free space, caching can make the normally small overhead of thin provisioning very small to unmeasurable. Q. Will migration to SSD zoning vs. HDD based block/pages impact data compression? A. It shouldn’t, since compression is done at a level where zoning isn’t an issue. Compression is only applicable to blocks or files. Q. Does compressing blocks on computational storage devices have the disadvantage of not reducing the PCIe bandwidth since raw data has to be transferred over to the storage devices? A. Yes. But the same is true of any storage device; so computational storage is no worse in respect of the transfer of the data, but it provides much more apparent storage on the device once it gets there. A computational storage device requires no application changes to do this. Q. How do we measure performance in out-of- line <data> reduction? A. Data reduction techniques like compression and deduplication can be done in-line (that is, while writing the data) or out-of-line (as a later point in time). Out-of-line shifts the compute required from now—where big horsepower is required if there’s to be no impact on storage performance, to later, where smaller processors can take their time. Out-of-line data reduction requires more space to store the data, as it’s unreduced when it’s written. These tradeoffs also have impacts on performance (both back-end latency and bandwidth). This all impacts the total cost of the system. It’s not so much that we need to measure the performance of in-line vs. out-of-line, something we know how to do, and declare one a winner; but it’s whether the system provides us the needed performance at the right cost. That’s a purchasing decision, not a technology one. Q. How do customers (or vendors) decide how wide their deduplication net should be, i.e. one disk, per file, across one file system, one storage system, or multiple storage systems? A. By testing and balancing the savings vs. the cost. One thing is true: the balance right now is very definitely in favor of deduplicating at every level where possible. Vendors can demonstrate huge space savings advantages by doing so. Consumers, as indicated by my answer to the previous question, need to look at the whole system and its cost vs. performance, and buy on that basis. Q. Is compression like doing deduplication on a very small and very local scale? A. You could think of it as bit-level deduplication, and then realize that you can stretch an analogy to breaking point… Q. Are some blocks or files so small that it’s not worth doing deduplication or cloning because the extra metadata will be larger than the block/file space savings? A. Yes. They’re often stored as is – but they do need metadata to say that they’re raw and not reduced. Q. Do cloning and snapshots operate only at the block level or can they operate at the file or object level too? A. Cloning and snapshots can operate at the file or object level, as long as there is an efficient way of extracting and storing the differences. Sometimes it’s cheaper and simpler just to copy the whole thing, especially for small files or objects. Q. Why does (Virtual Data Optimizer) VDO do dedupe before compression if the other way is preferable? Why is it better to compress then deduplicate? A. That’s a decision that the designers of VDO felt gave them the best storage efficiencies and reasonable compute overheads. (It’s also not the only system that uses this order.) But the dedupe scope of VDO is relatively small. Compression then deduplication allows in-line compression with out-of-line and much broader deduplication across very large sets of data, and there are many systems that use this order for that reason. Q. There’s also so much stuff because we (as an industry) have enabled storing so much stuff. (cheaply/affordably) Today’s business and storage market would look and act differently if costs were different. Data reduction’s interaction with encryption (e.g. proper ordering) could be useful to mention. Or a topic for another presentation! A. We’ll consider it! Remember I said we were taking a deeper dive on the topic of data reduction? We have two more webcast in this series – one on compression and the other on data deduplication. You can access them here:

• Compression: Putting the Squeeze on Storage – Available on-demand
• Not Again! Data Deduplication for Storage Systems – Live on November 10, 2020, on-demand after that date

Almost 800 people have already watched our webcast “Optimizing NVMe over Fabrics Performance with Different Ethernet Transports: Host Factors” where SNIA experts covered the factors impacting different Ethernet transport performance for NVMe over Fabrics (NVMe-oF) and provided data comparisons of NVMe over Fabrics tests with iWARP, RoCEv2 and TCP. If you missed the live event, watch it on-demand at your convenience.

The session generated a lot of questions, all answered here in this blog. In fact, many of the questions have prompted us to continue this discussion with future webcasts on NVMe-oF performance. Please follow us on Twitter @SNIANSF for upcoming dates.

Q. What factors will affect the performance of NVMe over RoCEv2 and TCP when the network between host and target is longer than typical Data Center environment? i.e., RTT > 100ms

A. For a large deployment with long distance, congestion management and flow control will be the most critical considerations to make sure performance is guaranteed. In a very large deployment, network topology, bandwidth subscription to storage target, and connection ratio are all important factors that will impact the performance of NVMe-oF.

Q. Were the RoCEv2 tests run on 'lossless' Ethernet and the TCP tests run on 'lossy' Ethernet?

A. Both iWARP and RoCEv2 tests were run in a back to back configuration without a switch in the middle, but with Link Flow Control turned on.

Q. Just to confirm, this is with pure ROCEv2? No TCP, right? ROCEv2 end 2 end (initiator 2 target)?

A. Yes, for RoCEv2 test, that was RoCEv2 Initiator to RoCEv2 target.

Q. How are the drives being preconditioned? Is it based on I/O size or MTU size? 

A. Storage is pre-conditioned by I/O size and type of the selected workload. MTU size is not relevant.  The selected workload is applied until performance changes are time invariant - i.e. until performance stabilizes within a range known as steady state.  Generally, the workload is tracked by specific I/O size and type to remain within a data excursion of 20% and a slope of 10%.

Q. Are the 6 SSDs off a single Namespace, or multiple? If so, how many Namespaces used?

A. Single namespace.

Q. What I/O generation tool was used for the test?

A. Calypso CTS IO Stimulus generator which is based on libaio. CTS has same engine as fio and applies IOs to the block IO level.  Note vdbench and iometer are java-based file system level and higher in the software stack.

Q. Given that NVMe SSD performance is high with low latency, is it not that the performance bottleneck is shifted to the storage controller?

A. Test I/Os are applied to the logical storage seen by host on the target server in our attempt to normalize the host and target in order to assess NIC-Wire-NIC performance. The storage controller is beneath this layer and not applicable to this test. If we test the storage directly on the target - not over the wire - then we can see impact of the controller and controller related issues (such as garbage collection, over provisioning, table structures, etc.)

Q. What are the specific characteristics of RoCEv2 that restrict it to 'rack' scale deployments?  In other words, what is restricting it from larger scale deployments?

A. RoCEv2 can, and does, scale beyond the rack if you have one of three things:

1. A lossless network with DCB (priority flow control)
2. Congestion management with solutions like ECN
3. Newer RoCEv2-capable adapters that support out of order packet receive and selective re-transmission

Your mileage will vary based upon features of different network vendors.

Q. Is there an option to use some caching mechanism on host side?

A. Host side has RAM cache per platform set up but is held constant among these tests. 

Q. Was there caching in the host?

A. The test used host memory for NVMe over Fabrics.

Q. Were all these topics from the description covered?  In particular, #2?
We will cover the variables:

1. How many CPU cores are needed (I’m willing to give)?
2. Optane SSD or 3D NAND SSD?
3. How deep should the Q-Depth be?
4. Why do I need to care about MTU?

A. Cores - see TC/QD sweep to see optimal OIO.  Core Usage/Required can be inferred from this. Note incongruity of TC/QD to OIO 8, 16, 32, 48 in this case.  

1. The test used a dual socket server on target with Intel^Ò Xeon^Ò Platinum 8280L processor with 28 cores. Target server only used one processor so that all the workloads were on a single NUMA node. 1-4% CPU utilization is the average of 28 cores.
2. SSD-1 is Optane SSD, SSD-2 is 3D NAND.
3. Normally QD is set to 32.
4. You do not need to care about MTU, at least in our test, we saw minimal performance differences.

Q. The result of 1~4% of CPU utilization on target is based on single SSD? Do you expect to see much higher CPU utilization if the amount of SSD increases?

A. CPU % is the target server for the 6 SSD LUN.

Q. Is there any difference between the different transports and the sensitivity of lost packets?

A. Theoretically, iWARP and TCP are more tolerant to packet lost. iWARP is based on TCP/IP, TCP provides flow control and congestion management that can still perform in a congested environment. In the event of packet loss, iWARP supports selective re-transmission and out of order packet receive, those technology can further improve the performance in a lossy network. While, RoCEv2 standard implementation does not tolerate packet loss and would require lossless network and would experience performance degradation when packet loss happens.

Q. 1. When you mean offload TCP, is this both at Initiator and target side or just host initiator side?
2. Do you see any improvement with ADQ on TCP?

A. RDMA iWARP in the test has a complete offload TCP engine on the network adapter on both Initiator and target side. Application Device Queues (ADQ) can significantly improve throughput, latency and most importantly latency jitter with dedicated CPU core allocated for NVMe-oF solutions.

Q. Since the CPU utilization is extremely low on the host, any comments about the CPU role in NVMe-oF and the impact of offloading?

A. NVMe-oF was designed to reduce the CPU load on target as shown in the test. On the initiator side CPU load will be a little bit higher. RDMA, as an offloaded technology, requires fairly minimal CPU utilization. NVMe over TCP still uses TCP stack in the kernel to do all the work, thus CPU still plays an important role. Also, the test was done with a high-end Intel^Ò Xeon^Ò  Processor with very powerful processing capability, if a processor with less processing power is used, CPU utilization would be higher.

Q. 1. What should be the ideal incapsulated data (inline date) size for best performance in a real-world scenario? 2. How could one optimize buffer copies at block level in NVMe-oF?

A. 1. There is no simple answer to this question. The impact of incapsulated data size to performance in the real-world scenario is more complicated as switch is playing a critical role in the whole network. Whether there is a shallow buffer switch or a deep buffer switch, switch settings like policy, congestion management etc. would all impact the overall performance. 2. There are multiple explorations to improve the performance of NVMe-oF by reducing or optimizing buffer copies. One possible option is to use controller memory buffer introduced in NVMe Specification 1.2.

Q. Is it possible to combine any of the NVMe-of technologies with SPDK - user space processing?

A. SPDK currently supports all these Ethernet-based transports: iWarp, RoCEv2 and TCP.

Q. You indicated that TCP is non-offloaded, but doesn't it still use the 'pseudo-standard' offloads like Checksum, LSO, RSS, etc?  It just doesn't have the entire TCP stack offloaded?

A. Yes, stateless offloads are supported and used.

Q. What is the real idea in using 4 different SSDs? Why didn't you use 6 or 8 or 10? What is the message you are trying to relay? I understand that SSD1 is higher/better performing than SSD2.

A. We used a six SSD LUN in both SSD-1 and SSD-2.  We compared higher performance - lower capacity Optane to lower performance - higher capacity NVMe.  Note NVMe is 10X capacity of Optane.

Q. It looks like one of the key takeaways is that SSD specs matter. Can you explain (without naming brands) the main differences between SSD-1 and SSD-2?

A. Manufacturer specs are only a starting point and actual performance depends on the workload.  Large differences are seen for small block RND W workloads and large block SEQ R workloads.

Q. What is the impact to the host CPU and memory during the tests? Wondering what minimum CPU and memory are necessary to achieve peak NVMe-oF performance, which leads to describe how much application workload one might be able to achieve.

A. The test did not limit CPU core or memory to try the minimal configuration to achieve peak NVMe-oF performance. This might be an interesting topic we can cover in the future presentation.  (We measured target server CPU usage, not host / initiator CPU Usage).

Q. Did you let the tests run for 2 hours and then take results? (basically, warm up the cache/SSD characterization)?

A. We precondition with the TC/QD Sweep test then run the remaining 3 tests back to back to take advantage of the preconditioning done in the first test.

Q. How do you check outstanding IOs?

A. We use OIO = TC x QD in test settings and populate each thread with the QD jobs. We do not look at in flight OIO, but wait for all OIOs to complete and measure response times.

Q. Where can we get the performance test specifications as defined by SNIA?

A. You can find the test specification on the SNIA website here.

Q. Have these tests been run using FC-NVMe. If so, how did they fare?

A. We have not yet run tests your NVMe over Fibre Channel.

Q. What tests did you use? FIO, VDBench, IOZone, or just DD or IOMeter? What was the CPU peak utilization? and what CPUs did you use?

A. CTS IO generator which is similar to fio as both are based on libaio and test at the block level.  Vdbench, iozone and Iometer are java file system level.  DD is direct and lacks complex scripting.  Fio allows compiles scripting but not multiple variables per loop - i.e. requires iterative tests and post-test compilation vs. CTS which has multi variable - multi loop concurrency.

Q. What test suites did you use for testing?

A. Calypso CTS tests

Q. I heard that iWARP is dead?

A. No, iWARP is not dead. There are multiple Ethernet network adapter vendors supporting iWARP now. The adapter used in the test supports iWARP, RoCEv2 and TCP at the same time.

Q. Can you post some recommendation on the switch setup and congestion?

A. The test talked about in this presentation used back to back configuration without switch. We will have a presentation in the near future to take into account switch settings and will share more information at that time. Don’t forget to follow us on Twitter @SNIANSF for dates of upcoming webcasts.

Almost 800 people have already watched our webcast “Optimizing NVMe over Fabrics Performance with Different Ethernet Transports: Host Factors” where SNIA experts covered the factors impacting different Ethernet transport performance for NVMe over Fabrics (NVMe-oF) and provided data comparisons of NVMe over Fabrics tests with iWARP, RoCEv2 and TCP. If you missed the live event, watch it on-demand at your convenience. The session generated a lot of questions, all answered here in this blog. In fact, many of the questions have prompted us to continue this discussion with future webcasts on NVMe-oF performance. Please follow us on Twitter @SNIANSF for upcoming dates. Q. What factors will affect the performance of NVMe over RoCEv2 and TCP when the network between host and target is longer than typical Data Center environment? i.e., RTT > 100ms A. For a large deployment with long distance, congestion management and flow control will be the most critical considerations to make sure performance is guaranteed. In a very large deployment, network topology, bandwidth subscription to storage target, and connection ratio are all important factors that will impact the performance of NVMe-oF. Q. Were the RoCEv2 tests run on ‘lossless’ Ethernet and the TCP tests run on ‘lossy’ Ethernet? A. Both iWARP and RoCEv2 tests were run in a back to back configuration without a switch in the middle, but with Link Flow Control turned on. Q. Just to confirm, this is with pure ROCEv2? No TCP, right? ROCEv2 end 2 end (initiator 2 target)? A. Yes, for RoCEv2 test, that was RoCEv2 Initiator to RoCEv2 target. Q. How are the drives being preconditioned? Is it based on I/O size or MTU size?  A. Storage is pre-conditioned by I/O size and type of the selected workload. MTU size is not relevant.  The selected workload is applied until performance changes are time invariant – i.e. until performance stabilizes within a range known as steady state.  Generally, the workload is tracked by specific I/O size and type to remain within a data excursion of 20% and a slope of 10%. Q. Are the 6 SSDs off a single Namespace, or multiple? If so, how many Namespaces used? A. Single namespace. Q. What I/O generation tool was used for the test? A. Calypso CTS IO Stimulus generator which is based on libaio. CTS has same engine as fio and applies IOs to the block IO level.  Note vdbench and iometer are java-based file system level and higher in the software stack. Q. Given that NVMe SSD performance is high with low latency, is it not that the performance bottleneck is shifted to the storage controller? A. Test I/Os are applied to the logical storage seen by host on the target server in our attempt to normalize the host and target in order to assess NIC-Wire-NIC performance. The storage controller is beneath this layer and not applicable to this test. If we test the storage directly on the target – not over the wire – then we can see impact of the controller and controller related issues (such as garbage collection, over provisioning, table structures, etc.) Q. What are the specific characteristics of RoCEv2 that restrict it to ‘rack’ scale deployments? In other words, what is restricting it from larger scale deployments? A. RoCEv2 can, and does, scale beyond the rack if you have one of three things:

1. A lossless network with DCB (priority flow control)
2. Congestion management with solutions like ECN
3. Newer RoCEv2-capable adapters that support out of order packet receive and selective re-transmission

Your mileage will vary based upon features of different network vendors. Q. Is there an option to use some caching mechanism on host side? A. Host side has RAM cache per platform set up but is held constant among these tests. Q. Was there caching in the host? A. The test used host memory for NVMe over Fabrics. Q. Were all these topics from the description covered?  In particular, #2? We will cover the variables:

1. How many CPU cores are needed (I’m willing to give)?
2. Optane SSD or 3D NAND SSD?
3. How deep should the Q-Depth be?
4. Why do I need to care about MTU?

A. Cores – see TC/QD sweep to see optimal OIO.  Core Usage/Required can be inferred from this. Note incongruity of TC/QD to OIO 8, 16, 32, 48 in this case.

1. The test used a dual socket server on target with Intel^Ò Xeon^Ò Platinum 8280L processor with 28 cores. Target server only used one processor so that all the workloads were on a single NUMA node. 1-4% CPU utilization is the average of 28 cores.
2. SSD-1 is Optane SSD, SSD-2 is 3D NAND.
3. Normally QD is set to 32.
4. You do not need to care about MTU, at least in our test, we saw minimal performance differences.

Q. The result of 1~4% of CPU utilization on target is based on single SSD? Do you expect to see much higher CPU utilization if the amount of SSD increases? A. CPU % is the target server for the 6 SSD LUN. Q. Is there any difference between the different transports and the sensitivity of lost packets? A. Theoretically, iWARP and TCP are more tolerant to packet lost. iWARP is based on TCP/IP, TCP provides flow control and congestion management that can still perform in a congested environment. In the event of packet loss, iWARP supports selective re-transmission and out of order packet receive, those technology can further improve the performance in a lossy network. While, RoCEv2 standard implementation does not tolerate packet loss and would require lossless network and would experience performance degradation when packet loss happens. Q. 1. When you mean offload TCP, is this both at Initiator and target side or just host initiator side? 2. Do you see any improvement with ADQ on TCP? A. RDMA iWARP in the test has a complete offload TCP engine on the network adapter on both Initiator and target side. Application Device Queues (ADQ) can significantly improve throughput, latency and most importantly latency jitter with dedicated CPU core allocated for NVMe-oF solutions. Q. Since the CPU utilization is extremely low on the host, any comments about the CPU role in NVMe-oF and the impact of offloading? A. NVMe-oF was designed to reduce the CPU load on target as shown in the test. On the initiator side CPU load will be a little bit higher. RDMA, as an offloaded technology, requires fairly minimal CPU utilization. NVMe over TCP still uses TCP stack in the kernel to do all the work, thus CPU still plays an important role. Also, the test was done with a high-end Intel^Ò Xeon^Ò  Processor with very powerful processing capability, if a processor with less processing power is used, CPU utilization would be higher. Q. 1. What should be the ideal incapsulated data (inline date) size for best performance in a real-world scenario? 2. How could one optimize buffer copies at block level in NVMe-oF? A. 1. There is no simple answer to this question. The impact of incapsulated data size to performance in the real-world scenario is more complicated as switch is playing a critical role in the whole network. Whether there is a shallow buffer switch or a deep buffer switch, switch settings like policy, congestion management etc. would all impact the overall performance. 2. There are multiple explorations to improve the performance of NVMe-oF by reducing or optimizing buffer copies. One possible option is to use controller memory buffer introduced in NVMe Specification 1.2. Q. Is it possible to combine any of the NVMe-of technologies with SPDK – user space processing? A. SPDK currently supports all these Ethernet-based transports: iWarp, RoCEv2 and TCP. Q. You indicated that TCP is non-offloaded, but doesn’t it still use the ‘pseudo-standard’ offloads like Checksum, LSO, RSS, etc?  It just doesn’t have the entire TCP stack offloaded? A. Yes, stateless offloads are supported and used. Q. What is the real idea in using 4 different SSDs? Why didn’t you use 6 or 8 or 10? What is the message you are trying to relay? I understand that SSD1 is higher/better performing than SSD2. A. We used a six SSD LUN in both SSD-1 and SSD-2.  We compared higher performance – lower capacity Optane to lower performance – higher capacity NVMe.  Note NVMe is 10X capacity of Optane. Q. It looks like one of the key takeaways is that SSD specs matter. Can you explain (without naming brands) the main differences between SSD-1 and SSD-2? A. Manufacturer specs are only a starting point and actual performance depends on the workload. Large differences are seen for small block RND W workloads and large block SEQ R workloads. Q. What is the impact to the host CPU and memory during the tests? Wondering what minimum CPU and memory are necessary to achieve peak NVMe-oF performance, which leads to describe how much application workload one might be able to achieve. A. The test did not limit CPU core or memory to try the minimal configuration to achieve peak NVMe-oF performance. This might be an interesting topic we can cover in the future presentation.  (We measured target server CPU usage, not host / initiator CPU Usage). Q. Did you let the tests run for 2 hours and then take results? (basically, warm up the cache/SSD characterization)? A. We precondition with the TC/QD Sweep test then run the remaining 3 tests back to back to take advantage of the preconditioning done in the first test. Q. How do you check outstanding IOs? A. We use OIO = TC x QD in test settings and populate each thread with the QD jobs. We do not look at in flight OIO, but wait for all OIOs to complete and measure response times. Q. Where can we get the performance test specifications as defined by SNIA? A. You can find the test specification on the SNIA website here. Q. Have these tests been run using FC-NVMe. If so, how did they fare? A. We have not yet run tests your NVMe over Fibre Channel. Q. What tests did you use? FIO, VDBench, IOZone, or just DD or IOMeter? What was the CPU peak utilization? and what CPUs did you use? A. CTS IO generator which is similar to fio as both are based on libaio and test at the block level.  Vdbench, iozone and Iometer are java file system level.  DD is direct and lacks complex scripting.  Fio allows compiles scripting but not multiple variables per loop – i.e. requires iterative tests and post-test compilation vs. CTS which has multi variable – multi loop concurrency. Q. What test suites did you use for testing? A. Calypso CTS tests Q. I heard that iWARP is dead? A. No, iWARP is not dead. There are multiple Ethernet network adapter vendors supporting iWARP now. The adapter used in the test supports iWARP, RoCEv2 and TCP at the same time. Q. Can you post some recommendation on the switch setup and congestion? A. The test talked about in this presentation used back to back configuration without switch. We will have a presentation in the near future to take into account switch settings and will share more information at that time. Don’t forget to follow us on Twitter @SNIANSF for dates of upcoming webcasts.

The broad adoption of 5G, Internet of things (IoT) and edge computing will reshape the nature and role of enterprise and cloud storage over the next several years. What building blocks, capabilities and integration methods are needed to make this happen? That will be the topic of discussion at our live SNIA Cloud Storage Technologies webcast on October 21, 2020 “Storage Implications at the Velocity of 5G Streaming.” Join my SNIA expert colleagues, Steve Adams and Chip Maurer, for a discussion on common questions surrounding this topic, including: 

• With 5G, IoT and edge computing – how much data are we talking about?
• What will be the first applications leading to collaborative data-intelligence streaming?
• How can low latency microservices and AI quickly extract insights from large amounts of data?
• What are the emerging requirements for scalable stream storage – from peta to zeta?
• How do yesterday’s object-based batch analytic processing (Hadoop) and today’s streaming messaging capabilities (Apache Kafka and RabbitMQ) work together?
• What are the best approaches for getting data from the Edge to the Cloud?

I hope you will register today and join us on October 21^st. It’s live so please bring your questions!

The broad adoption of 5G, Internet of things (IoT) and edge computing will reshape the nature and role of enterprise and cloud storage over the next several years. What building blocks, capabilities and integration methods are needed to make this happen? That will be the topic of discussion at our live SNIA Cloud Storage Technologies webcast on October 21, 2020 “Storage Implications at the Velocity of 5G Streaming.” Join my SNIA expert colleagues, Steve Adams and Chip Maurer, for a discussion on common questions surrounding this topic, including: 

• With 5G, IoT and edge computing – how much data are we talking about?
• What will be the first applications leading to collaborative data-intelligence streaming?
• How can low latency microservices and AI quickly extract insights from large amounts of data?
• What are the emerging requirements for scalable stream storage – from peta to zeta?
• How do yesterday’s object-based batch analytic processing (Hadoop) and today’s streaming messaging capabilities (Apache Kafka and RabbitMQ) work together?
• What are the best approaches for getting data from the Edge to the Cloud?

I hope you will register today and join us on October 21^st. It’s live so please bring your questions!