Measuring power efficiency in datacenter storage is a complex endeavor. A number of factors play a role in assessing individual storage devices or system-level logical storage for power efficiency. Luckily, our SNIA experts make the measuring easier! In this SNIA Experts on Data blog series, our experts in the SNIA Solid State Storage Technical Work Group and the SNIA Green Storage Initiative explore factors to consider in power efficiency measurement, including the nature of application workloads, IO streams, and access patterns; the choice of storage products (SSDs, HDDs, cloud storage, and more); the impact of hardware and software components (host bus adapters, drivers, OS layers); and access to read and write caches, CPU and GPU usage, and DRAM utilization. Join us on our journey to better power efficiency as we continue with Part 2: Impact of Workloads on Power Efficiency Measurement.  And if you missed Part 1: Key Issues in Power Efficiency Measurement, you can find it here.  Bookmark this blog  and check back in March and April for the continuation of our four-part series. And explore the topic further in the SNIA Green Storage Knowledge Center. Part 2: Impact of Workloads on Power Efficiency Measurement Workloads are a significant driving force behind power consumption in computing systems. Different tasks and applications place diverse demands on hardware, leading to fluctuations in the amount of power used. Here’s a breakdown of how workloads can influence power consumption:

• CPU Utilization. The CPU’s power consumption increases as it processes tasks, with more demanding workloads that involve complex calculations or multitasking leading to higher CPU utilization and, consequently, elevated power usage.
• Memory Access is another key factor. Accessing memory modules consumes power, and workloads that heavily rely on frequent memory read and write operations can significantly contribute to increased power consumption.
• Disk Activity, particularly read and write operations on storage devices (whether HDDs or SSDs), consumes power. Workloads that involve frequent data access or large file transfers can lead to an uptick in power consumption. GPU Usage plays a crucial role, especially in tasks like gaming, video editing, and machine learning. High GPU utilization for rendering complex graphics or training deep neural networks can result in substantial power consumption.
• Network Communication tasks, such as data transfers, streaming, or online gaming, require power from both the CPU and the network interface. The extent of communication and data throughput can significantly affect overall power usage.
• In devices equipped with displays, Screen Brightness directly impacts power consumption. Brighter screens consume more power, which means workloads involving continuous display usage contribute to higher power consumption.
• I/O Operations encompass interactions with peripherals like storage devices or printers. These operations can lead to short bursts of power consumption, especially if multiple devices are connected.
• Understanding the contrast between Idle and Active States is essential. Different workloads can transition devices between these states, with idle periods generally exhibiting lower power consumption. However, certain workloads may keep components active even during seemingly idle times.
• Dynamic Voltage and Frequency Scaling are prevalent in many systems, allowing them to adjust the voltage and frequency of components based on workload demands. Increased demand leads to higher clock speeds and voltage, ultimately resulting in more significant power consumption.
• Background Processes also come into play. Background applications, updates, and system maintenance tasks can impact power consumption, even when the user isn’t actively engaging with the device.

In practical terms, comprehending how various workloads affect power consumption is vital for optimizing energy efficiency. For instance, laptops can extend their battery life by reducing screen brightness, closing unnecessary applications, and selecting power-saving modes. Moreover, SSDs are designed with optimizations for background processes in mind. Garbage collection and NAND Flash cell management often occur during idle periods or periods of low-impact workloads. Likewise, data centers and cloud providers strategically manage workloads to minimize energy consumption and operational costs while upholding performance standards. The post Power Efficiency Measurement – Our Experts Make It Clear – Part 2 first appeared on SNIA Compute, Memory and Storage Blog.

72% of organizations have encountered incompatibility issues between various object storage implementations according to a poll at our recent SNIA Cloud Storage Technologies Initiative webinar, “Navigating the Complexities of Object Storage Compatibility.” If you missed the live presentation or you would like to see the answers to the other poll questions we asked the audience, you can view it on-demand at the SNIA Educational Library. The audience was highly-engaged during the live event and asked several great questions. Here are answers to them all. Q. Do you see the need for fast object storage for AI kind of workloads? A. Yes, the demand for fast object storage in AI workloads is growing. Initially, object storage was mainly used for backup or archival purposes. However, its evolution into Data Lakes and the introduction of features like the S3 SELECT API have made it more suitable for data analytics. The launch of Amazon's S3 Express, a faster yet more expensive tier, is a clear indication of this trend. Other vendors are following suit, suggesting a shift towards object storage as a primary data storage platform for specific workloads. Q. As Object Storage becomes more prevalent in the primary storage space, could you talk about data protection, especially functionalities like synchronous replication and multi-site deployments - or is your view that this is not needed for object storage deployments? A. Data protection, including functionalities like synchronous replication and multi-site deployments, is essential for object storage, especially as it becomes more prevalent in primary storage. Various object storage implementations address this differently. For instance, Amazon S3 supports asynchronous replication. Azure ZRS (Zone-redundant storage) offers something akin to synchronous replication within a specific geographical area. Many on-premises solutions provide multi-site deployment and replication capabilities. It's crucial for vendors to offer distinct features and value additions, giving customers a range of choices to best meet their specific requirements. Ultimately, customers must define their data availability and durability needs and select the solution that aligns with their use case. Q. Regarding polling question #3 during the webinar, why did the question only ask “above 10PB?” We look for multi-PB like 100PB ... does this mean Object Storage is not suitable for multi PB? A. Object storage is inherently scalable and can support deployments ranging from petabyte to exabyte scale. However, scalability can vary based on specific implementations. Each object storage solution may have its own limits in terms of capacity. It's important to review the details of each solution to ensure it meets your specific needs for multi-petabyte scale deployments. Q. Is Wasabi 100% Compatible with Amazon S3? A. While we typically avoid discussing specific vendors in a general forum, it's important to note that most 'S3-compatible' object storage implementations have some discrepancies when compared to Amazon S3. These differences can vary in significance. Therefore, we always recommend testing your actual workload against the specific object storage solution to identify any critical issues or incompatibilities. Q. What are the best ways to see a unified view of different types of storage -- including objects, file and blocks? This may be most relevant for enterprise-wide data tracking and multi-cloud deployments. A. There are various solutions available from different vendors that offer visibility into multiple types of storage, including object, file, and block storage. These solutions are particularly useful for enterprise-wide data management and multi-cloud deployments. However, this topic extends beyond the scope of our current discussion. SNIA might consider addressing this in a separate, dedicated webinar in the future. Q. Is there any standard object storage implementation against which the S3 compatibility would be defined? A. Amazon S3 serves as a de facto standard for object storage implementation. Independent software vendors (ISVs) can decide the degree of compatibility they want to achieve with Amazon S3, including which features to implement and to what extent. The objective isn't necessarily to achieve identical functionality across all implementations, but rather for each ISV to be cognizant of the specific differences and potential incompatibilities in their own solutions. Being aware of these discrepancies is key, even if complete compatibility isn't achieved. Q. With the introduction of directory buckets, do you anticipate vendors picking up compatibility there as well or maintaining a strictly flat namespace? A. That's an intriguing question. We are putting together an on-going object storage forum, which will delve into more in follow-up calls, and will serve as a platform for these kinds of discussions. We anticipate addressing not only the concept of directory buckets versus a flat namespace, but also exploring other ideas like performance enhancements and alternate transport layers for S3. This forum is intended to be a collaborative space for discussing future directions in object storage. If you’re interested, contact the cloudtwg@snia.org. Q. How would an incompatibility be categorized as something that is important for clients vs. just something that doesn't meet the AWS spec/behavior? A. Incompatibilities should be assessed based on the specific needs and priorities of each implementor. While we don't set universal compatibility goals, it's up to every implementor to determine how closely they align with S3 or other protocols. They must decide whether to address any discrepancies in behavior or functionality based on their own objectives and their clients' requirements. Essentially, the significance of an incompatibility is determined by its impact on the implementor's goals and client needs. Q. Have customers experienced incompatibilities around different SDKs with regard to HA behaviors? Load balancers vs. round robin DNS vs. other HA techniques on-prem and in the cloud? A. Yes, customers do encounter incompatibilities related to different SDKs, particularly concerning high availability (HA) behaviors. Object storage encompasses more than just APIs; it also involves implementation choices like load balancing decisions and HA techniques. Discrepancies often arise due to these differences, especially when object storage solutions are deployed within a customer's data center and need to integrate with the existing networking infrastructure. These incompatibilities can be due to various factors, including whether load balancing is handled through round robin DNS, dedicated load balancers, or other HA techniques, either on-premises or in the cloud. Q. Any thoughts on keeping pace with AWS as they evolve the S3 API? I'm specifically thinking about the new Directory Bucket type and the associated API changes to support hierarchy. A. We at the SNIA Cloud Storage Technical Work Group are in dialogue with Amazon and are encouraging their participation in our planned Plugfest at SDC’24. Their involvement would be invaluable in helping us anticipate upcoming changes and understand new developments, such as the Directory Bucket type and its associated API changes. This new variation of S3 from Amazon, which differs from the original implementation, underscores the importance of compatibility testing. While complete compatibility may not always be achievable, it's crucial for ISVs to be fully aware of how their implementations differ from S3's evolving standards. Q. When it comes to object store data protection with backup software, do you see also some incompatibilities with recovered data? A. When data is backed up to an object storage system, there's a fundamental expectation that it can be reliably retrieved later. This reliability is a cornerstone of any storage platform. However, issues can arise when data is initially stored in one specific object storage implementation and later transferred to a different one. If this transfer isn't executed in accordance with the backup software provider's requirements, it could lead to difficulties in accessing the data in the future. Therefore, careful planning and adherence to recommended practices are crucial during any data migration process to prevent such compatibility issues. The SNIA Cloud Storage Technical Work Group is actively working on this topic. If you want to get involved, reach out at cloudtwg@snia.org and follow us @sniacloud_com

72% of organizations have encountered incompatibility issues between various object storage implementations according to a poll at our recent SNIA Cloud Storage Technologies Initiative webinar, “Navigating the Complexities of Object Storage Compatibility.” If you missed the live presentation or you would like to see the answers to the other poll questions we asked the audience, you can view it on-demand at the SNIA Educational Library. The audience was highly-engaged during the live event and asked several great questions. Here are answers to them all. Q. Do you see the need for fast object storage for AI kind of workloads? A. Yes, the demand for fast object storage in AI workloads is growing. Initially, object storage was mainly used for backup or archival purposes. However, its evolution into Data Lakes and the introduction of features like the S3 SELECT API have made it more suitable for data analytics. The launch of Amazon’s S3 Express, a faster yet more expensive tier, is a clear indication of this trend. Other vendors are following suit, suggesting a shift towards object storage as a primary data storage platform for specific workloads. Q. As Object Storage becomes more prevalent in the primary storage space, could you talk about data protection, especially functionalities like synchronous replication and multi-site deployments – or is your view that this is not needed for object storage deployments? A. Data protection, including functionalities like synchronous replication and multi-site deployments, is essential for object storage, especially as it becomes more prevalent in primary storage. Various object storage implementations address this differently. For instance, Amazon S3 supports asynchronous replication. Azure ZRS (Zone-redundant storage) offers something akin to synchronous replication within a specific geographical area. Many on-premises solutions provide multi-site deployment and replication capabilities. It’s crucial for vendors to offer distinct features and value additions, giving customers a range of choices to best meet their specific requirements. Ultimately, customers must define their data availability and durability needs and select the solution that aligns with their use case. Q. Regarding polling question #3 during the webinar, why did the question only ask “above 10PB?” We look for multi-PB like 100PB … does this mean Object Storage is not suitable for multi PB? A. Object storage is inherently scalable and can support deployments ranging from petabyte to exabyte scale. However, scalability can vary based on specific implementations. Each object storage solution may have its own limits in terms of capacity. It’s important to review the details of each solution to ensure it meets your specific needs for multi-petabyte scale deployments. Q. Is Wasabi 100% Compatible with Amazon S3? A. While we typically avoid discussing specific vendors in a general forum, it’s important to note that most ‘S3-compatible’ object storage implementations have some discrepancies when compared to Amazon S3. These differences can vary in significance. Therefore, we always recommend testing your actual workload against the specific object storage solution to identify any critical issues or incompatibilities. Q. What are the best ways to see a unified view of different types of storage — including objects, file and blocks? This may be most relevant for enterprise-wide data tracking and multi-cloud deployments. A. There are various solutions available from different vendors that offer visibility into multiple types of storage, including object, file, and block storage. These solutions are particularly useful for enterprise-wide data management and multi-cloud deployments. However, this topic extends beyond the scope of our current discussion. SNIA might consider addressing this in a separate, dedicated webinar in the future. Q. Is there any standard object storage implementation against which the S3 compatibility would be defined? A. Amazon S3 serves as a de facto standard for object storage implementation. Independent software vendors (ISVs) can decide the degree of compatibility they want to achieve with Amazon S3, including which features to implement and to what extent. The objective isn’t necessarily to achieve identical functionality across all implementations, but rather for each ISV to be cognizant of the specific differences and potential incompatibilities in their own solutions. Being aware of these discrepancies is key, even if complete compatibility isn’t achieved. Q. With the introduction of directory buckets, do you anticipate vendors picking up compatibility there as well or maintaining a strictly flat namespace? A. That’s an intriguing question. We are putting together an on-going object storage forum, which will delve into more in follow-up calls, and will serve as a platform for these kinds of discussions. We anticipate addressing not only the concept of directory buckets versus a flat namespace, but also exploring other ideas like performance enhancements and alternate transport layers for S3. This forum is intended to be a collaborative space for discussing future directions in object storage. If you’re interested, contact the cloudtwg@snia.org. Q. How would an incompatibility be categorized as something that is important for clients vs. just something that doesn’t meet the AWS spec/behavior? A. Incompatibilities should be assessed based on the specific needs and priorities of each implementor. While we don’t set universal compatibility goals, it’s up to every implementor to determine how closely they align with S3 or other protocols. They must decide whether to address any discrepancies in behavior or functionality based on their own objectives and their clients’ requirements. Essentially, the significance of an incompatibility is determined by its impact on the implementor’s goals and client needs. Q. Have customers experienced incompatibilities around different SDKs with regard to HA behaviors? Load balancers vs. round robin DNS vs. other HA techniques on-prem and in the cloud? A. Yes, customers do encounter incompatibilities related to different SDKs, particularly concerning high availability (HA) behaviors. Object storage encompasses more than just APIs; it also involves implementation choices like load balancing decisions and HA techniques. Discrepancies often arise due to these differences, especially when object storage solutions are deployed within a customer’s data center and need to integrate with the existing networking infrastructure. These incompatibilities can be due to various factors, including whether load balancing is handled through round robin DNS, dedicated load balancers, or other HA techniques, either on-premises or in the cloud. Q. Any thoughts on keeping pace with AWS as they evolve the S3 API? I’m specifically thinking about the new Directory Bucket type and the associated API changes to support hierarchy. A. We at the SNIA Cloud Storage Technical Work Group are in dialogue with Amazon and are encouraging their participation in our planned Plugfest at SDC’24. Their involvement would be invaluable in helping us anticipate upcoming changes and understand new developments, such as the Directory Bucket type and its associated API changes. This new variation of S3 from Amazon, which differs from the original implementation, underscores the importance of compatibility testing. While complete compatibility may not always be achievable, it’s crucial for ISVs to be fully aware of how their implementations differ from S3’s evolving standards. Q. When it comes to object store data protection with backup software, do you see also some incompatibilities with recovered data? A. When data is backed up to an object storage system, there’s a fundamental expectation that it can be reliably retrieved later. This reliability is a cornerstone of any storage platform. However, issues can arise when data is initially stored in one specific object storage implementation and later transferred to a different one. If this transfer isn’t executed in accordance with the backup software provider’s requirements, it could lead to difficulties in accessing the data in the future. Therefore, careful planning and adherence to recommended practices are crucial during any data migration process to prevent such compatibility issues. The SNIA Cloud Storage Technical Work Group is actively working on this topic. If you want to get involved, reach out at cloudtwg@snia.org and follow us @sniacloud_com

Any discussion about storage systems is incomplete without the mention of Throughput, IOPs, and Latency. But what exactly do these terms mean, and why are they important? To answer these questions, the SNIA Networking Storage Forum (NSF) is bringing back our popular webinar series, “Everything You Wanted to Know About Storage, But Were Too Proud to Ask.” Collectively, these three terms are often referred to as storage performance metrics. Performance can be defined as the effectiveness of a storage system to address I/O needs of an application or workload. Different application workloads have different I/O patterns, and with that arises different bottlenecks, so there is no “one-size fits all” in storage systems. These storage performance metrics help with storage solution design and selection based on application/workload demands. Join us on February 7, 2024, for “Everything You Wanted to Know About Throughput, IOPS, and Latency, But Were Too Proud to Ask.” In this webinar, we’ll cover:

• What storage performance metrics mean – understanding key terminology nuances
• Why users/storage administrators should care about them
• How these metrics impact application performance
• Real-world use cases

It’s a live event and our experts will be available to answer any questions you may have. I hope you’ll register today. If you are not familiar with our “Too Proud to Ask” series, I encourage you to check out these 11 great on-demand webinars that provide 101 lessons on a wide range of storage terminology.

Any discussion about storage systems is incomplete without the mention of Throughput, IOPs, and Latency. But what exactly do these terms mean, and why are they important? To answer these questions, the SNIA Networking Storage Forum (NSF) is bringing back our popular webinar series, “Everything You Wanted to Know About Storage, But Were Too Proud to Ask.” Collectively, these three terms are often referred to as storage performance metrics. Performance can be defined as the effectiveness of a storage system to address I/O needs of an application or workload. Different application workloads have different I/O patterns, and with that arises different bottlenecks, so there is no “one-size fits all” in storage systems. These storage performance metrics help with storage solution design and selection based on application/workload demands. Join us on February 7, 2024, for “Everything You Wanted to Know About Throughput, IOPS, and Latency, But Were Too Proud to Ask.” In this webinar, we’ll cover:

• What storage performance metrics mean – understanding key terminology nuances
• Why users/storage administrators should care about them
• How these metrics impact application performance
• Real-world use cases

It’s a live event and our experts will be available to answer any questions you may have. I hope you’ll register today. If you are not familiar with our “Too Proud to Ask” series, I encourage you to check out these 11 great on-demand webinars that provide 101 lessons on a wide range of storage terminology. The post Here’s Everything You Wanted to Know About Throughput, IOPs, and Latency first appeared on SNIA on Network Storage.

Measuring power efficiency in datacenter storage is a complex endeavor. A number of factors play a role in assessing individual storage devices or system-level logical storage for power efficiency. Luckily, our SNIA experts make the measuring easier! In this SNIA Experts on Data blog series, our experts in the SNIA Solid State Storage Technical Work Group and the SNIA Green Storage Initiative explore factors to consider in power efficiency measurement, including the nature of application workloads, IO streams, and access patterns; the choice of storage products (SSDs, HDDs, cloud storage, and more); the impact of hardware and software components (host bus adapters, drivers, OS layers); and access to read and write caches, CPU and GPU usage, and DRAM utilization. Join us on our journey to better power efficiency as we begin with Part 1: Key Issues in Power Efficiency Measurement. Bookmark this blog and check back in February, March, and April for the continuation of our four-part series. And explore the topic further in the SNIA Green Storage Knowledge Center. Part 1: Key Issues in Power Efficiency Measurement Ensuring accurate and precise power consumption measurements is challenging, especially at the individual device level, where even minor variations can have a significant impact. Achieving reliable data necessitates addressing factors like calibration, sensor quality, and noise reduction. Furthermore, varying workloads in systems require careful consideration to accurately capture transient power spikes and average power consumption. Modern systems are composed of interconnected components that affect each other’s power consumption, making it difficult to isolate individual component power usage. The act of measuring power itself consumes energy, creating a trade-off between measurement accuracy and the disturbance caused by measurement equipment. To address this, it’s important to minimize measurement overheads while still obtaining meaningful data. Environmental factors such as temperature, humidity, and airflow, can unpredictably influence power consumption, emphasizing the need for standardized test environments. Rapid workload changes can lead to transient power behavior that may require specialized equipment for accurate measurement. Software running on a system significantly influences power consumption, emphasizing the importance of selecting representative workloads and ensuring consistent software setups across measurements. Dynamic voltage and frequency scaling are used in many systems to optimize power consumption, and understanding their effects under different conditions is crucial. Correctly interpreting raw power consumption data is essential to draw meaningful conclusions about efficiency. This requires statistical analysis and context-specific considerations. Real-world variability, stemming from manufacturing differences, component aging, and user behavior, must also be taken into account in realistic assessments. Addressing these challenges necessitates a combination of precise measurement equipment, thoughtful experimental design, and a deep understanding of the system and device being investigated. In our next blog, Part 2, we will examine the impact of workloads on power efficiency measurement. The post Power Efficiency Measurement – Our Experts Make It Clear – Part 1 first appeared on SNIA Compute, Memory and Storage Blog.

Workloads using generative artificial intelligence trained on large language models are frequently throttled by insufficient resources (e.g., memory, storage, compute or network dataflow bottlenecks). If not identified and addressed, these dataflow bottlenecks can constrain Gen AI application performance well below optimal levels. Given the compelling uses across natural language processing (NLP), video analytics, document resource development, image processing, image generation, and text generation, being able to run these workloads efficiently has become critical to many IT and industry segments. The resources that contribute to generative AI performance and efficiency include CPUs, DPUs, GPUs, FPGAs, plus memory and storage controllers. On January 24, 2024, the SNIA Networking Storage Forum (NSF) is convening a panel of experts for a discussion on how to tackle Gen AI challenges at our live webinar, “Accelerating Generative AI – Options for Conquering the Dataflow Bottlenecks,” where a broad cross-section of industry veterans will provide insight into the following:

• Defining the Gen AI dataflow bottlenecks
• Tools and methods for identifying acceleration options
• Matchmaking the right xPU solution to the target Gen AI workload(s)
• Optimizing the network to support acceleration options
• Moving data closer to processing, or processing closer to data
• The role of the software stack in determining Gen AI performance

This is a session you don’t want to miss! Register today to save your spot. The post Accelerating Generative AI first appeared on SNIA on Network Storage.

The SNIA Cloud Storage Technologies Initiative (CSTI) recently hosted a live webinar, “Simplified Namespace Management – The Open Standards Way,” where David Slik, Chair of the SNIA Cloud Storage Technical Work Group (TWG) provided a fascinating overview of how to tackle the complexities of dynamic namespaces. If you missed the live webinar, you can view it on-demand and access a copy of the webinar slides at the SNIA Educational Library. Attendees at the live event asked several interesting questions. Here are answers to them all. Q. How are the queues assigned to individual namespaces? How many queues are assigned for a particular namespace, can we customize it and if so, how? What is the difference between normal namespace and SR-IOV enabled namespace? Can you please explain sets domain and endurance group? A. For NVMe^® namespace management, the Cloud Storage TWG recommends the use of the SNIA Swordfish® Specification. The SNIA Cloud Data Management Interface (CDMI™) is designed to provide a cloud abstraction that hides the complexities and details of the underlying storage implementation and infrastructure from the cloud user. Hiding storage implementation and infrastructure complexities are a key part of what makes a cloud a cloud, in that: a) knowledge and specification of the underlying infrastructure should not be required in order to consume storage services (simplification of use), b) the underlying infrastructure will be constantly and dynamically changing, and these changes should not be visible to the end user (to enable operational flexibility for the cloud provider), and, c) the desired quality of service, data protection, and other client-required storage services should be indicated by intent, rather than as a side-effect of the underlying configuration (ideally via declarative metadata). As these are also three key principles of CDMI. This guides us to avoid directly exposing or managing the underlying storage infrastructure. Q. How do the responses behave if the responses are really large - i.e. Can I get just the metadata that might warn me there are 70 billion objects at the top level and I don’t really want to spend the time to get them all before deciding or diving into one of them? A. When obtaining information about data objects and containers, CDMI allows the request to specify which fields should be returned in the JSON response. This is described in sections 8.4.1 and 9.4.1 of the CDMI specification, and uses standard URI query parameters. For example, to get the object name, metadata and number of children for a container, the following request URI would be used: GET /cdmi/2.0.0/MyContainer/?objectName&metadata&childrenrange CDMI also allows range requests for listing a subset of the children of a container. For example, listing the first 200 children of a container would be accomplished by using the following request URI: GET /cdmi/2.0.0/MyContainer/?children=0­-199 There also is a draft extension to CDMI to support recursive children listing and obtaining information about children in a single request, which can dramatically reduce the number of requests required to enumerate a container when information about each child is required. Q. Where can I go to get help using CDMI with my application? A. SNIA provides free access to the CDMI specification on the SNIA website. Extensions to the CDMI standard are also publicly available here.The SNIA Cloud Storage (TWG) provides implementation assistance and discusses extensions and errata to the standard as part of its weekly work group calls. Interested parties are encouraged to join the TWG. Q. If I were not using CDMI, what tools or methods would I need to incorporate to do the same kind of operations? What else is out there?  A. The Cloud Storage TWG does not know of any similar standards for namespace management. In order to manage namespaces without using CDMI, one would need to do the following: a) Define or select an HTTP-based protocol that provides basic request/response semantics and includes authentication. This is provided by all of the cloud providers for their cloud APIs. b) Define or select a set of APIs for enumerating namespaces, for example, the ListBuckets API in AWS S3, and the Azure Files List Directories and Files API in Microsoft Azure. c) Define a set of APIs for listing and specifying how namespaces (files, directories, objects and containers) can be exported or imported. While each of these exists for the major cloud providers, they are unique for each provider and storage type. CDMI provides a common, open and unified way to manage all types of storage namespaces. Q. How does CDMI help address security in my namespace management? A. CDMI provides a number of security functions that assist with namespace management: a) Every object in CDMI, including namespaces, can have an access control list (ACL) that specifies what operations can be performed against that object. This is described in section 17.2 of the CDMI specification. ACLs are based on standard NFSv4 ACLs, and allow metadata modifications (E.g. CDMI exports and CDMI imports) to have separate access control entries (ACEs). b) CDMI objects can have their access control decisions delegated to a customer-provided system via Delegated Access Control (DAC), which can provide finer-grained access control than ACLs where needed, as needed. This allows policies to take into account the specific import and export requests themselves, and to interface with policy enforcement frameworks such as XACML and open source policy engines such as the Open Policy Agent (OPA). c) CDMI allows mapping of user credentials to the user principal and group to be performed by external systems, such as Active Directory. This mapping can be on an object-by-object basis, allowing objects managed by different security domains to co-exist within a single unified namespace.  

The SNIA Cloud Storage Technologies Initiative (CSTI) recently hosted a live webinar, “Simplified Namespace Management – The Open Standards Way,” where David Slik, Chair of the SNIA Cloud Storage Technical Work Group (TWG) provided a fascinating overview of how to tackle the complexities of dynamic namespaces. If you missed the live webinar, you can view it on-demand and access a copy of the webinar slides at the SNIA Educational Library. Attendees at the live event asked several interesting questions. Here are answers to them all. Q. How are the queues assigned to individual namespaces? How many queues are assigned for a particular namespace, can we customize it and if so, how? What is the difference between normal namespace and SR-IOV enabled namespace? Can you please explain sets domain and endurance group? A. For NVMe^® namespace management, the Cloud Storage TWG recommends the use of the SNIA Swordfish® Specification. The SNIA Cloud Data Management Interface (CDMI) is designed to provide a cloud abstraction that hides the complexities and details of the underlying storage implementation and infrastructure from the cloud user. Hiding storage implementation and infrastructure complexities are a key part of what makes a cloud a cloud, in that: a) knowledge and specification of the underlying infrastructure should not be required in order to consume storage services (simplification of use), b) the underlying infrastructure will be constantly and dynamically changing, and these changes should not be visible to the end user (to enable operational flexibility for the cloud provider), and, c) the desired quality of service, data protection, and other client-required storage services should be indicated by intent, rather than as a side-effect of the underlying configuration (ideally via declarative metadata). As these are also three key principles of CDMI. This guides us to avoid directly exposing or managing the underlying storage infrastructure. Q. How do the responses behave if the responses are really large – i.e. Can I get just the metadata that might warn me there are 70 billion objects at the top level and I don’t really want to spend the time to get them all before deciding or diving into one of them? A. When obtaining information about data objects and containers, CDMI allows the request to specify which fields should be returned in the JSON response. This is described in sections 8.4.1 and 9.4.1 of the CDMI specification, and uses standard URI query parameters. For example, to get the object name, metadata and number of children for a container, the following request URI would be used: GET /cdmi/2.0.0/MyContainer/?objectName&metadata&childrenrange CDMI also allows range requests for listing a subset of the children of a container. For example, listing the first 200 children of a container would be accomplished by using the following request URI: GET /cdmi/2.0.0/MyContainer/?children=0­-199 There also is a draft extension to CDMI to support recursive children listing and obtaining information about children in a single request, which can dramatically reduce the number of requests required to enumerate a container when information about each child is required. Q. Where can I go to get help using CDMI with my application? A. SNIA provides free access to the CDMI specification on the SNIA website. Extensions to the CDMI standard are also publicly available here.The SNIA Cloud Storage (TWG) provides implementation assistance and discusses extensions and errata to the standard as part of its weekly work group calls. Interested parties are encouraged to join the TWG. Q. If I were not using CDMI, what tools or methods would I need to incorporate to do the same kind of operations? What else is out there?  A. The Cloud Storage TWG does not know of any similar standards for namespace management. In order to manage namespaces without using CDMI, one would need to do the following: a) Define or select an HTTP-based protocol that provides basic request/response semantics and includes authentication. This is provided by all of the cloud providers for their cloud APIs. b) Define or select a set of APIs for enumerating namespaces, for example, the ListBuckets API in AWS S3, and the Azure Files List Directories and Files API in Microsoft Azure. c) Define a set of APIs for listing and specifying how namespaces (files, directories, objects and containers) can be exported or imported. While each of these exists for the major cloud providers, they are unique for each provider and storage type. CDMI provides a common, open and unified way to manage all types of storage namespaces. Q. How does CDMI help address security in my namespace management? A. CDMI provides a number of security functions that assist with namespace management: a) Every object in CDMI, including namespaces, can have an access control list (ACL) that specifies what operations can be performed against that object. This is described in section 17.2 of the CDMI specification. ACLs are based on standard NFSv4 ACLs, and allow metadata modifications (E.g. CDMI exports and CDMI imports) to have separate access control entries (ACEs). b) CDMI objects can have their access control decisions delegated to a customer-provided system via Delegated Access Control (DAC), which can provide finer-grained access control than ACLs where needed, as needed. This allows policies to take into account the specific import and export requests themselves, and to interface with policy enforcement frameworks such as XACML and open source policy engines such as the Open Policy Agent (OPA). c) CDMI allows mapping of user credentials to the user principal and group to be performed by external systems, such as Active Directory. This mapping can be on an object-by-object basis, allowing objects managed by different security domains to co-exist within a single unified namespace.  

Object storage has firmly established itself as a cornerstone of modern data centers and cloud infrastructure. Ensuring API compatibility has become crucial for object storage developers who want to benefit from the wide ecosystem of existing applications. However, achieving compatibility can be challenging due to the complexity and variety of the APIs, access control mechanisms, and performance and scalability requirements. The SNIA Cloud Storage Technologies Initiative, together with the SNIA Cloud Storage Technical Work Group, is working to address the issues of cloud object storage complexity and interoperability. We’re kicking off 2024 with two exciting initiatives: 1) a webinar on June 9, 2024, and 2) a Plugfest in September of 2024. Here are the details: Webinar: Navigating the Complexities of Object Storage Compatibility In this webinar, we'll highlight real-world incompatibilities found in various object storage implementations. We'll discuss specific examples of existing discrepancies, such as missing or incorrect response headers, unsupported API calls, and unexpected behavior. We’ll also describe the implications these have on actual client applications. This analysis is based on years of experience with implementation, deployment, and evaluation of a wide range of object storage systems on the market. Attendees will leave with a deeper understanding of the challenges around compatibility and how to address them in their own applications. Register here to join us on January 9, 2024. Plugfest: Cloud Object Storage Plugfest SNIA is planning an open collaborative Cloud Object Storage Plugfest co-located at SNIA Storage Developer Conference (SDC) scheduled for September 2024 to work on improving cross-implementation compatibility for client and/or server implementations of private and public cloud object storage solutions. This endeavor is designed to be an independent, vendor-neutral effort with broad industry support, focused on a variety of solutions, including on-premises and in the cloud. This Plugfest aims to reduce compatibility issues, thus improving customer experience and increasing the adoption rate of object storage solutions. Click here to let us know if you're interested. We hope you will consider participating in both of these initiatives!