When it comes to selecting the right NVMe over Fabrics™ (NVMe-oF™) solution, one should look beyond test results that demonstrate NVMe-oF’s dramatic reduction in latency and consider the other, more important, questions such as “How does the transport really impact application performance?” and “How does the transport holistically fit into my environment?”

To date, the focus has been on specialized fabrics like RDMA (e.g., RoCE) because it provides the lowest possible latency, as well as Fibre Channel because it is generally considered to be the most reliable.  However, with the introduction of NVMe-oF/TCP this conversation must be expanded to also include considerations regarding scale, cost, and operations. That’s why the SNIA Networking Storage Forum (NSF) is hosting a webcast series that will dive into answering these questions beyond the standard answer “it depends.”

The first in this series will be on March 25, 2021 “NVMe-oF: Looking Beyond Performance Hero Numbers” where SNIA experts with deep NVMe and fabric technology expertise will discuss the thought process you can use to determine pros and cons of a fabric for your environment, including:

• Use cases driving fabric choices  
• NVMe transports and their strengths
• Industry dynamics driving adoption
• Considerations for scale, security, and efficiency

Future webcasts will dive deeper and cover operating and managing NVMe-oF, discovery automation, and securing NVMe-oF. I hope you will register today. Our expert panel will be available on March 25^th to answer your questions.

When it comes to selecting the right NVMe over Fabrics™ (NVMe-oF™) solution, one should look beyond test results that demonstrate NVMe-oF’s dramatic reduction in latency and consider the other, more important, questions such as “How does the transport really impact application performance?” and “How does the transport holistically fit into my environment?” To date, the focus has been on specialized fabrics like RDMA (e.g., RoCE) because it provides the lowest possible latency, as well as Fibre Channel because it is generally considered to be the most reliable. However, with the introduction of NVMe-oF/TCP this conversation must be expanded to also include considerations regarding scale, cost, and operations. That’s why the SNIA Networking Storage Forum (NSF) is hosting a webcast series that will dive into answering these questions beyond the standard answer “it depends.” The first in this series will be on March 25, 2021 “NVMe-oF: Looking Beyond Performance Hero Numbers” where SNIA experts with deep NVMe and fabric technology expertise will discuss the thought process you can use to determine pros and cons of a fabric for your environment, including:

• Use cases driving fabric choices
• NVMe transports and their strengths
• Industry dynamics driving adoption
• Considerations for scale, security, and efficiency

Future webcasts will dive deeper and cover operating and managing NVMe-oF, discovery automation, and securing NVMe-oF. I hope you will register today. Our expert panel will be available on March 25^th to answer your questions.

So much of what we discuss within SNIA is the latest emerging technologies in storage. While it’s good to know about what technology is coming, it’s also important to understand the technologies that should be sunsetted.

It’s the topic of our next SNIA Networking Storage Forum (NSF) webcast on February 3, 2021, “Storage Technologies & Practices Ripe for Refresh.”  In this webcast, you’ll learn about storage technologies and practices in your data center that are ready for refresh or possibly retirement. Find out why some long-standing technologies and practices should be re-evaluated. We’ll discuss:

• Obsolete hardware, protocols, interfaces and other aspects of storage
• Why certain technologies are no longer in general use
• Technologies on their way out and why
• Drivers for change
• Justifications for obsoleting proven technologies
• Trade-offs risks: new faster/better vs. proven/working tech

Register today and bring your questions for our panel of experts. 

So much of what we discuss within SNIA is the latest emerging technologies in storage. While it’s good to know about what technology is coming, it’s also important to understand the technologies that should be sunsetted. It’s the topic of our next SNIA Networking Storage Forum (NSF) webcast on February 3, 2021, “Storage Technologies & Practices Ripe for Refresh.”  In this webcast, you’ll learn about storage technologies and practices in your data center that are ready for refresh or possibly retirement. Find out why some long-standing technologies and practices should be re-evaluated. We’ll discuss:

• Obsolete hardware, protocols, interfaces and other aspects of storage
• Why certain technologies are no longer in general use
• Technologies on their way out and why
• Drivers for change
• Justifications for obsoleting proven technologies
• Trade-offs risks: new faster/better vs. proven/working tech

Register today and bring your questions for our panel of experts.

2020 brought new developments in persistent memory and computational storage. SNIA Compute, Memory, and Storage Initiative was pleased to sponsor two tracks at the recent Flash Memory Summit where industry leaders captured the advances.  Videos and presentations are now available.

In the Persistent Memory Track, Dave Eggleston of Intuitive Cognition Consulting and Chris Petersen of Facebook combine to deliver a state of the union address for the industry effort underway to deliver persistent memory. They examine industry advances of persistent memory media, the new devices and form factors for persistent memory attachment, remote and direct-attached PM with low latency interfaces like CXL, and describe the best fit applications and use cases for persistent memory.

Jia Shi of Oracle and Yao Yue of Twitter then dive into a rapid-fire presentation on two examples of how persistent memory is changing the landscape – in appliances, in infrastructure, and in applications - from the perspective of a social networking company and a cloud and enterprise software provider.  They highlight the motivation for using persistent memory and the delivered results

Finally, Ginger Gilsdorf of Intel and Tom Coughlin of Coughlin Associates look ahead to how Persistent Memory technology is evolving, including maximizing performance in next-generation applications, and provide their perspective on PM market growth projections.

The track concludes with speakers reuniting in a panel to discuss the reasons that have stopped persistent memory from gaining wider usage and identifying breakthroughs that are beginning to appear.

The Computational Storage Track opens with an update by Chuck Sobey of Channel Science who discusses the shifting of compute power to the storage; use cases including database, big data, AI/ML, and edge applications; and how the framework for computational storage is driven by SNIA and the NVM Express standards groups.

Stephen Bates of Eideticom follows with an outline of the state of the nation in computational storage standards. He then describes computational storage examples already in use that illustrate ways storage challenges are being met, and comments on promising directions to explore for the future.

Andy Walls of IBM then discusses using computational storage to handle big data, allowing data to reside close to processing power, thus allowing processing tasks to be in-line with data accesses. He covers computational storage examples already in use for application distribution and other promising directions to explore for the future.

Neil Werdmuller and Jason Molgaard of Arm discuss flexible computational storage solutions, and how data-driven applications that benefit from database searches, data manipulation, and machine learning can perform better and be more scalable if developers add computation directly to storage.

A lively panel with Arm, Eideticom, NGD Systems, and ScaleFlux rounds out the track, discussing keys to making computational storage work in your applications.  

Enjoy these presentations and contact us at askcmsi@snia.org with your questions and comments!

2020 brought new developments in persistent memory and computational storage. SNIA Compute, Memory, and Storage Initiative was pleased to sponsor two tracks at the recent Flash Memory Summit where industry leaders captured the advances.  Videos and presentations are now available. In the Persistent Memory Track, Dave Eggleston of Intuitive Cognition Consulting and Chris Petersen of Facebook combine to deliver a state of the union address for the industry effort underway to deliver persistent memory. They examine industry advances of persistent memory media, the new devices and form factors for persistent memory attachment, remote and direct-attached PM with low latency interfaces like CXL, and describe the best fit applications and use cases for persistent memory. Jia Shi of Oracle and Yao Yue of Twitter then dive into a rapid-fire presentation on two examples of how persistent memory is changing the landscape – in appliances, in infrastructure, and in applications – from the perspective of a social networking company and a cloud and enterprise software provider.  They highlight the motivation for using persistent memory and the delivered results Finally, Ginger Gilsdorf of Intel and Tom Coughlin of Coughlin Associates look ahead to how Persistent Memory technology is evolving, including maximizing performance in next-generation applications, and provide their perspective on PM market growth projections. The track concludes with speakers reuniting in a panel to discuss the reasons that have stopped persistent memory from gaining wider usage and identifying breakthroughs that are beginning to appear. The Computational Storage Track opens with an update by Chuck Sobey of Channel Science who discusses the shifting of compute power to the storage; use cases including database, big data, AI/ML, and edge applications; and how the framework for computational storage is driven by SNIA and the NVM Express standards groups. Stephen Bates of Eideticom follows with an outline of the state of the nation in computational storage standards. He then describes computational storage examples already in use that illustrate ways storage challenges are being met, and comments on promising directions to explore for the future. Andy Walls of IBM then discusses using computational storage to handle big data, allowing data to reside close to processing power, thus allowing processing tasks to be in-line with data accesses. He covers computational storage examples already in use for application distribution and other promising directions to explore for the future. Neil Werdmuller and Jason Molgaard of Arm discuss flexible computational storage solutions, and how data-driven applications that benefit from database searches, data manipulation, and machine learning can perform better and be more scalable if developers add computation directly to storage. A lively panel with Arm, Eideticom, NGD Systems, and ScaleFlux rounds out the track, discussing keys to making computational storage work in your applications. Enjoy these presentations and contact us at askcmsi@snia.org with your questions and comments! The post Experts Speak at Flash Memory Summit first appeared on SNIA Compute, Memory and Storage Blog.

The COVID-19 Pandemic has amplified cybersecurity concerns, particularly related to the cloud. Threat actors have recognized a unique opportunity to exploit pandemic-related vulnerabilities through social engineering attacks, business email compromise, work from home, and other remote weak points. This results in increased risk and occurrence of ransomware attacks and data breaches that can disrupt or totally compromise organizations’ ability to conduct business. Partnering with a cybersecurity company Melbourne can help businesses strengthen their defenses against these evolving threats. These security incidents can also subject victims to liability for violations of privacy and data breach notification laws. Then there’s the issue of facing allegations that could impact your career and personal life. Internet-related offences are increasingly common, and many individuals are caught up in investigations without realising the full implications. For anyone dealing with online fraud, hacking accusations, or other cyber-related charges, having an experienced internet crimes attorney who understands digital laws and defence strategies can be the key to securing a favourable outcome. The right legal support can help you navigate these complex cases effectively. The SNIA Cloud Storage Technologies Initiative (CSTI) will be taking on this important topic with a live webcast on January 20, 2021, “Data Privacy and Data Protection in the COVID Era” where our SNIA experts will discuss:

• The changing threat landscape due to COVID-19
• Common security failures and their consequences
• Recent attacker exploits
• Data protection concerns:
  • Strategies to combat malware
  • Minimizing ransomware risks
• How emerging technologies (5G, IoT, AI, etc.) expand the threat landscape

Register today to learn key considerations to mitigate the cybersecurity risks resulting from the COVID pandemic.

The COVID-19 Pandemic has amplified cybersecurity concerns particularly related to the cloud. Threat actors have recognized a unique opportunity to exploit pandemic-related vulnerabilities through social engineering attacks, business email compromise, work from home and other remote weak points. This results in increased risk and occurrence of ransomware attacks and data breaches that can disrupt or totally compromise organizations’ ability to conduct business. These security incidents can also subject victims to liability for violations of privacy and data breach notification laws. The SNIA Cloud Storage Technologies Initiative (CSTI) will be taking on this important topic with a live webcast on January 20, 2021, “Data Privacy and Data Protection in the COVID Era” where our SNIA experts will discuss:

• The changing threat landscape due to COVID-19
• Common security failures and their consequences
• Recent attacker exploits
• Data protection concerns:
  • Strategies to combat malware
  • Minimizing ransomware risks
• How emerging technologies (5G, IoT, AI, etc.) expand the threat landscape

Register today to learn key considerations to mitigate the cybersecurity risks resulting from the COVID pandemic.

The SNIA Networking Storage Forum (NSF) recently took on the topics surrounding data reduction with a 3-part webcast series that covered Data Reduction Basics, Data Compression and Data Deduplication. If you missed any of them, they are all available on-demand.

In Not Again! Data Deduplication for Storage Systems” our SNIA experts discussed how to reduce the number of copies of data that get stored, mirrored, or backed up. Attendees asked some interesting questions during the live event and here are answers to them all.

Q. Why do we use the term rehydration for deduplication?  I believe the use of the term rehydration when associated with deduplication is misleading. Rehydration is the activity of bringing something back to its original content/size as in compression. With deduplication the action is more aligned with a scatter/gather I/O profile and this does not require rehydration.

A. "Rehydration" is used to cover the reversal of both compression and deduplication. It is used more often to cover the reversal of compression, though there isn't a popularly-used term to specifically cover the reversal of deduplication (such as "re-duplication").  When reading compressed data, if the application can perform the decompression then the storage system does not need to decompress the data, but if the compression was transparent to the application then the storage (or backup) system will decompress the data prior to letting the application read it. You are correct that deduplicated files usually remain in a deduplicated state on the storage when read, but the storage (or backup) system recreates the data for the user or application by presenting the correct blocks or files in the correct order.

Q. What is the impact of doing variable vs fixed block on primary storage Inline?

A. Deduplication is a resource intensive process. The process of sifting the data inline by anchoring, fingerprinting and then filtering for duplicates not only requires high computational resources, but also adds latency on writes. For primary storage systems that require high performance and low latencies, it is best to keep these impacts of dedupe low. Doing dedupe with variable-sized blocks or extents (for e.g. with Rabin fingerprinting) is more intensive than using simple fixed-sized blocks. However, variable-sized segmentation is likely to give higher storage efficiency in many cases. Most often this tradeoff between latency/performance vs. storage efficiency tips in favor of applying simpler fixed-size dedupe in primary storage systems.

Q. Are there special considerations for cloud storage services like OneDrive?

A. As far as we know, Microsoft OneDrive avoids uploading duplicate files that have the same filename, but does not scan file contents to deduplicate identical files that have different names or different extensions. As with many remote/cloud backup or replication services, local deduplication space savings do not automatically carry over to the remote site unless the entire volume/disk/drive is replicated to the remote site at the block level. Please contact Microsoft or your cloud storage provider for more details about any space savings technology they might use.

Q. Do we have an error rate calculation system to decide which type of deduplication we use?

A. The choice of deduplication technology to use largely depends on the characteristics of the dataset and the environment in which deduplication is done. For example, if the customer is running a performance and latency sensitive system for primary storage purposes, then the cost of deduplication in terms of the resources and latencies incurred may be too high and the system may use very simple fixed-size block based dedupe. However, if the system/environment allows for spending extra resources for the sake of storage efficiency, then a more complicated variable-sized extent based dedupe may be used. About error rates themselves, a dedupe storage system should always be built with strong cryptographic hash-based fingerprinting so that the error rates of collisions are extremely low. Errors due to collisions in a dedupe system may lead to data loss or corruption, but as mentioned earlier these can be avoided by using strong cryptographic functions.

Q. Considering the current SSD QLC limitations and endurance... Can we say that if a right choice for deduped storage?

A. In-line deduplication either has no effect or reduces the wear on NAND storage because less data is written. Post-process deduplication usually increases wear on NAND storage because blocks are written then later erased--due to deduplication--and the space later fills with new data. If the system uses post-process deduplication, then the storage software or storage administrator needs to weigh the space savings benefits vs. the increased wear on NAND flash. Since QLC NAND is usually less expensive and has lower write endurance than SLC/MLC/TLC NAND, one might be less likely to use post-process deduplication on QLC NAND than on more expensive NAND which has higher endurance levels.

Q. On slides 11/12 - why not add compaction as well - "fitting" the data onto respective blocks and "if 1k file, not leaving the rest 3k of 4k block empty"?

A. We covered compaction in our webcast on data reduction basics “Everything You Wanted to Know About Storage But Were Too Proud to Ask: Data Reduction.” See slide #18 below.

Again, I encourage you to check out this Data Reduction series and follow us on Twitter @SNIANSF for dates and topics of more SNIA NSF webcasts.

The SNIA Networking Storage Forum (NSF) recently took on the topics surrounding data reduction with a 3-part webcast series that covered Data Reduction Basics, Data Compression and Data Deduplication. If you missed any of them, they are all available on-demand. In Not Again! Data Deduplication for Storage Systems” our SNIA experts discussed how to reduce the number of copies of data that get stored, mirrored, or backed up. Attendees asked some interesting questions during the live event and here are answers to them all. Q. Why do we use the term rehydration for deduplication?  I believe the use of the term rehydration when associated with deduplication is misleading. Rehydration is the activity of bringing something back to its original content/size as in compression. With deduplication the action is more aligned with a scatter/gather I/O profile and this does not require rehydration. A. “Rehydration” is used to cover the reversal of both compression and deduplication. It is used more often to cover the reversal of compression, though there isn’t a popularly-used term to specifically cover the reversal of deduplication (such as “re-duplication”).  When reading compressed data, if the application can perform the decompression then the storage system does not need to decompress the data, but if the compression was transparent to the application then the storage (or backup) system will decompress the data prior to letting the application read it. You are correct that deduplicated files usually remain in a deduplicated state on the storage when read, but the storage (or backup) system recreates the data for the user or application by presenting the correct blocks or files in the correct order. Q. What is the impact of doing variable vs fixed block on primary storage Inline? A. Deduplication is a resource intensive process. The process of sifting the data inline by anchoring, fingerprinting and then filtering for duplicates not only requires high computational resources, but also adds latency on writes. For primary storage systems that require high performance and low latencies, it is best to keep these impacts of dedupe low. Doing dedupe with variable-sized blocks or extents (for e.g. with Rabin fingerprinting) is more intensive than using simple fixed-sized blocks. However, variable-sized segmentation is likely to give higher storage efficiency in many cases. Most often this tradeoff between latency/performance vs. storage efficiency tips in favor of applying simpler fixed-size dedupe in primary storage systems. Q. Are there special considerations for cloud storage services like OneDrive? A. As far as we know, Microsoft OneDrive avoids uploading duplicate files that have the same filename, but does not scan file contents to deduplicate identical files that have different names or different extensions. As with many remote/cloud backup or replication services, local deduplication space savings do not automatically carry over to the remote site unless the entire volume/disk/drive is replicated to the remote site at the block level. Please contact Microsoft or your cloud storage provider for more details about any space savings technology they might use. Q. Do we have an error rate calculation system to decide which type of deduplication we use? A. The choice of deduplication technology to use largely depends on the characteristics of the dataset and the environment in which deduplication is done. For example, if the customer is running a performance and latency sensitive system for primary storage purposes, then the cost of deduplication in terms of the resources and latencies incurred may be too high and the system may use very simple fixed-size block based dedupe. However, if the system/environment allows for spending extra resources for the sake of storage efficiency, then a more complicated variable-sized extent based dedupe may be used. About error rates themselves, a dedupe storage system should always be built with strong cryptographic hash-based fingerprinting so that the error rates of collisions are extremely low. Errors due to collisions in a dedupe system may lead to data loss or corruption, but as mentioned earlier these can be avoided by using strong cryptographic functions. Q. Considering the current SSD QLC limitations and endurance… Can we say that if a right choice for deduped storage? A. In-line deduplication either has no effect or reduces the wear on NAND storage because less data is written. Post-process deduplication usually increases wear on NAND storage because blocks are written then later erased–due to deduplication–and the space later fills with new data. If the system uses post-process deduplication, then the storage software or storage administrator needs to weigh the space savings benefits vs. the increased wear on NAND flash. Since QLC NAND is usually less expensive and has lower write endurance than SLC/MLC/TLC NAND, one might be less likely to use post-process deduplication on QLC NAND than on more expensive NAND which has higher endurance levels. Q. On slides 11/12 – why not add compaction as well – “fitting” the data onto respective blocks and “if 1k file, not leaving the rest 3k of 4k block empty”? A. We covered compaction in our webcast on data reduction basics “Everything You Wanted to Know About Storage But Were Too Proud to Ask: Data Reduction.” See slide #18 below.

Again, I encourage you to check out this Data Reduction series and follow us on Twitter @SNIANSF for dates and topics of more SNIA NSF webcasts.