Last month, the SNIA Cloud Storage Technologies Community hosted a live webinar, “Community Driven S3 Compatibility Testing,” where engineers from multiple technology leaders shared their interoperability testing experiences among various S3 implementations at the SNIA Cloud Object Storage Plugfest in April 2025. The session was highly rated by the audience and generated several intriguing questions, which Frank Borich from IBM and Jeff Terrace from Google have answered here. 

Q: How does the community feel about the virtual host addressing versus path based addressing options for the API? Especially for on-prem customers/products this requires more complex DNS setups, and TLS certificates.

A: Virtual host addressing (e.g. bucketname.service.com) and path-style addressing (e.g. service.com/bucketname) both have tradeoffs.

As you allude to, virtual-host addressing makes DNS and SSL certificates more difficult. SSL also isn't possible for bucket names with dots in the name, since SSL certificate wildcards only support a single subdomain. But virtual-host style does allow for powerful DNS-based routing and load balancing, and it also segments web-based requests into different origins, which can be important for security.

Path-style addressing is simpler, but it doesn't allow for DNS-based features, and is currently on the deprecation path by AWS.

As a community, the reality is that both styles typically have to be supported by service implementers. Many tools and libraries make assumptions about one or the other being available.

Q: In what areas of the S3 API do you see the most varied implementations/behaviors? Suspended mode versioning?

A: We typically don't see any one area being different from others. The S3 API is composed of many different features, so what we've found is it's difficult to talk about a service being "S3 Compatible" or "S3 Interoperable" without breaking it down further. We've started brainstorming ideas about classifying the S3 API into concrete feature sets.

Q: From the IBM perspective chart: What is with the Noobaa implementation? This is an important development for us here and have a good feature list?

A: We are not able to rate the compliance level of any particular product at this point.

Q: Do you have any recommendation or best practice for developers of applications that require interoperability with object storage of different vendors? Are known differences documented somewhere? 

A: We'd recommend you reach out to each object storage provider for documentation on their interoperability. There's no central repository of providers and compatibility. As we mentioned, as part of the test tools effort, we are exploring the idea of categorizing interoperability into concrete feature sets that would make it easier to classify levels of compatibility between providers.

Q: Is there any abstraction layer for configuration that you could recommend? Is it preferable to try to adapt to the proprietary APIs or better to use S3 http interface for e.g. bucket policy configuration, which does not seem to be the promoted way of working for most vendors?

A: We would recommend differentiating your control plane and data plane usage of cloud storage providers. Data plane usage (i.e. reading and writing of objects) typically doesn't need to interact with bucket-level policies and configuration. We have seen many customers take advantage of generic configuration libraries such as Terraform, for managing control plane resources.

Q: Will the AWS S3 team participate in this effort? 

A: While we cannot speak for Amazon, we are hoping that AWS S3 will join our community. 

Q: How do you anticipate AWS S3 may benefit from this activity?  

A: We believe the SNIA Cloud Object Storage (COS) Plugfest community shares exactly the same “end customer excellence" goals as AWS S3. The SNIA COS Plugfest community is specifically designed to help improve overall industry interoperability, and make it easier to help application developers introduce and maintain capabilities.  

Q: Please say more about object storage interoperability; why is it important? 

A: We want to be very clear. There is no issue for clients working directly with AWS services. AWS S3 is a popular proprietary API, fully controlled and well managed by AWS. The area we are addressing is heterogeneous COS interoperability. We are doing this because, before third-party storage providers worked independently, repeating the same investment in test, but worked in silos, so no shared test results. Now that we can pool our resources among the Plugfest community, we can share test results under NDA, between participating companies. We believe this is important to address COS interoperability for configurations, including a number of third-party storage providers integrated across a number of ISV services, like backup, archive, disaster recovery and dynamic applications such as AI analytics, micro-services and container environments.   

Q: Who else should participate in this Plugfest effort? 

A: Again, we are focused on heterogeneous interoperability (including proprietary and open source projects); areas like on-prem, hybrid-cloud and multi-cloud solutions that are deployed at banks, health care, enterprise, government, retail, scientific / AI computing, etc. We are eager to expand COS Plugfest testing with multiple cloud service providers (CSPs), client and server providers, multi-protocol gateway providers, database vendors, and ISVs (including backup, archive and disaster recovery services), as well as tape archive solution vendors.   

Q: Will SNIA consider offering more remote COS Plugfest testing?  

A: Yes. We had a very good set of remote test sessions during the April ’25 COS Plugfest, and we are putting together plans now to enable more remote COS testing. We believe that face-to-face COS Plugfest engagement result in the very best collaboration, because we have immediate interaction to first identify where to test and analyze findings. This cuts weeks of delay from typical multi-company interactions and helps natively build the developer-to-developer network to expedite issues, as well as helps build the tribal community knowledge pool. However, we also see the need to extend the face-to-face network with an even broader set of remote contributors. This is to rapidly respond to new use cases as needed, expand the community, and accommodate the realities that not everyone can gain travel approval for every event. We see this especially in the area of over-seas travel.  Please contact us at askcloudplugfest@snia.org if you have questions and to join the Plugfest community to stay in touch with these important developments.  

Our next Cloud Object Storage Plugfest will be in Santa Clara, CA co-located with the SNIA Developer Conference (SDC’25), September 15-17, 2025. Learn how you and your company can be part of this Plugfest here.

The SNIA Cloud Storage Technologies (CST) community hosted a live webinar, “Building Community to Tackle Cloud Object Storage Incompatibilities,” where we highlighted how a multi-vendor group of industry innovators came together to address incompatibility issues between various object storage implementations that challenge many organizations.  The webinar, along with presentation slides, are available on-demand here.   

Webinar panelists included engineers from Dell, Google, Hammerspace, IBM, Vast, and Versity Software, all of whom shared their Plugfest experiences, and the audience asked several intriguing questions.  This is a compilation of responses from our guest speakers; thanks to all for their insights and contributions. 

Q. Will you have more SNIA Cloud Object Storage (COS) Plugfests in 2025? 

A. Yes, to participate in the next SNIA COS Plugfest, April 28-30, 2025, (co-located with One Day Regional SDC in Denver CO on April 30th), please pre-register at SNIA Cloud Object Storage Plugfest.  Our second SNIA COS Plugfest for 2025 will be hosted during SDC’25, Sept 15-17, 2025, in Santa Clara, CA.  If you have questions, please contact us at: askcloudplugfest@snia.org.

Note: A Plugfest is a collaborative developer event where industry experts come together, test their cloud object storage solutions, find problems, and fix them. SNIA takes care of the space and tools needed, and everyone agrees to keep issues resolved secret under NDA.

Q. Please provide examples of how end-user customers may be impacted due to unsupported API calls and unexpected behavior? 

A. There are many different ways interoperability issues are introduced that can impact user experience, such as introducing new features or functionality. Third-party object storage server implementations are always a bit behind on the latest changes, so from an application developers’ perspective, it's not always obvious what features are new. Unfortunately, with new changes, even basic functionality can break, like CRUDs (Create, Read, Update, and Delete) or other essentials such as request signing. 

An example was a feature which introduced new methods for validating data integrity with support for additional checksum algorithms. By changing behavior of SDK clients, previous stable workflows halted, with clients reporting 500 errors.  This was a more friendly example, because the breakage was fairly obvious. 

We've also seen this kind of thing happen when a third-party implementation ignores unrecognized HTTP request headers, which has been the default behavior for unrecognized HTTP headers since HTTP 1.0 (1996). One more non-obvious example of how introduction of new features (along with version skew) may impact customers is the introduction of headers that implement conditional writes, which are designed to prevent modification or deletion of objects if they do or do not exist. As an example, if a client were to enable a newly introduced conditional write header, which was not previously implemented by a third-party implementation, users could potentially experience unexpected behavior. 

This points to the need for more collaboration, where developers on both sides of the wire (end-to-end) ensure successful introduction of features, as well as explore how flexible their applications are to configure around feature changes or explore how to bake some of that flexibility into their application.  This is why it is critical to engage all developer stakeholders in open collaborative developer discussions. 

Q. Is there anything that end users should be looking out for?

A. Yes. This is why API versioning is so important when you encounter things like new headers and new checksums. Ideally, you should be able to look at the version of any widely used protocol and identify whether or not it’s expected to have that feature, or not. This has historically been a fundamental expectation of user-friendly protocols – they should mandate certain features and certain versions of the API to ensure basic functionality remains consistent and stable. It’s one method that helps address unexpected behavior while building consensus around interoperability.   

However, the current state of development allows different vendors to choose various API options in their respective implementations (these may not always match or overlap sufficiently to attain proper compatibility support). This allows, or even creates, situations where each implementation may work well within its own limited ecosystem, yet unexpected results may occur when configuring a variety of implementations within a new configuration or end-to-end solution. 

End users expect that everything should just work. They may not even be aware of the backing storage implementation or vendor. This is the primary motivation why organizations are eager to participate in the SNIA Cloud Object Storage Plugfest, not only to proactively broaden the set of their compatibility testing (to find bugs before customers find them) but also to gain agreement on industry best practices. 

Note: up until the formation of this SNIA Cloud Object Storage Plugfest community, there had been no vendor-neutral community available to help define an accepted set of protocol options. Until now, it had been up to separate developer teams to choose independently, and that’s exactly the issue we are resolving. We are organizing a method to help developer teams get on the same page at the same time.   

Q. My organization mandates seamless data portability between cloud instances. What do I need to do to make sure I am not locked into one solution?   

A. The goal of the SNIA Cloud Object Storage Plugfest is to bring the multi-vendor community together to help organizations ensure data portability. Our team recommends customers and end users encourage their cloud object storage providers to take part in this SNIA Cloud Object Storage Plugfest effort, to make sure products and services have gone through rigorous interoperability efforts and the hard work of testing against each other. 

Q. Amazon and Microsoft were notably missing in this webinar, do they plan to join this effort in the future? 

A. Microsoft actively participated in the inaugural SNIA Cloud Object Storage Plugfest in September at SDC’24. In fact, Microsoft brought a team including protocol experts who contributed to the Plugfest bug assessment and the Birds of a Feather (BOF) session, where we gathered insight to formulate next steps for the industry.  At the beginning of the Plugfest, Microsoft and Plugfest contributors quickly compared notes to identify trouble spots. Most everyone in the room knew right away what to look out for. There was a quick consensus on where to test, and the team invested time where needed. Microsoft was an integral part of this. 

AWS has been invited. We are continuing to welcome and encourage them to join the SNIA Cloud Object Storage Plugfest discussion.

Q. Ultimately, doesn’t AWS dictate S3?  If a consortium of vendors represented here added extensions to the API, or new self-describing API, wouldn’t this break things assuming AWS is the genesis of S3 API going forward? 

A. Yes, AWS S3 is a proprietary API for Cloud Object Storage, and any non-AWS implementation of S3 must conform with AWS S3 API and SDKs in order to interoperate. 

A similar situation existed in the late ‘90s, with Microsoft and the SMB (CIFS) protocol. At that time, Microsoft owned the client, server, SMB protocol, and dictated everything that happened.  Yet, an evolution occurred when Microsoft worked with SMB developers, including both proprietary and open source implementations. In 2008, Microsoft sponsored the SNIA CIFS/SMB Plugfest organized / hosted by SNIA as a trusted third-party storage industry association.  This event is still ongoing (now called SNIA SMB Interoperability Lab or “IOL”) and has been proven as a model for timely information exchange among Microsoft and the global ecosystem of SMB developers.  Note, Microsoft continues to own their IP, drives the pace and direction of its SMB protocol roadmap, including changes and release timeframes. 

We anticipate a similar working arrangement may be possible between SNIA and AWS (and others) as we expand community participation focused on multi-vendor, heterogeneous interoperability.   

Thanks to all the SNIA Cloud Object Storage Plugfest team for your time, effort and insights, 

Michael Hoard, SNIA, Chair for Cloud Storage Technologies (CST) community

The SNIA Cloud Storage Initiative hosted a live Webcast “Cloud Object Storage 101.” Like any “101” type course, there were a lot of good questions. Here they all are – with our answers. If you have additional questions, please let us know by commenting on this blog.

Q. How do you envision the new role of tape (LTO) in this unstructured data growth?

A. Exactly the same way that tape has always played a part; it’s the storage medium that requires no power to store cold data and is cheap per bit. Although it has a limited shelf life, and although we believe that flash will eventually replace it, it still has a secure & growing foreseeable future.

Q. What are your thoughts on whether object storage can exist outside the bounds of supporting file systems? Block devices directly storing objects using the key as reference and removing the intervening file system? A hierarchy of objects instead of files?

A. All of these things. Objects can be objects identified by an ID in a flat non-hierarchical structure; or we can impose a hierarchy by key- to objectID translation; or indeed, an object may contain complete file systems or be treated like a block device. There are really no restrictions on how we can build meta data that describes all these things over the bytes of storage that makes up an object.

Q. Can you run write insensitive low latency apps on object storage, ex: virtual machines?

A. Yes. Object storage can be made up of the same stuff as other high performance storage systems; for instance, flash connect via high bandwidth and low latency networks. Or they could even be object stores built over PCIe and NVDIMM.

Q. Is erasure coding (EC) expensive in terms of networking and resources utilization (especially in case of rebuild)?

A. No, that’s one of the advantages of EC. Rebuilds take place by reading data from many disks and writing it to many disks; in traditional RAID rebuilds, the focus is normally on the one disk that’s being rebuilt.

Q. Is there any overhead for small files or object use cases? Do you have a recommended size?

A. Each system will have its own advantages and disadvantages for objects of specific sizes. In general, object stores are designed to store billions of objects, so the number of objects is usually not an issue.

Q. Can you comment on Internet bandwidth limitations on geographically dispersed erasure coded data?

A. Smart caching can make a big difference, but at the end of the day, a geographically EC dispersed object store won’t be faster than a local store. You can’t beat the speed of light.

Q. The suppliers all claim easy exit strategies from their systems. If we were to use one of the on-premise solutions such as ECS or Cleversafe, and then down the road decide to move off-premise, is the migration/egress typically as easy as claimed?

A. In general, any proprietary interface might lock you in. The SNIA’s CDMI is vendor neutral, and supported by a number of vendors. Amazon’s S3 is a popular and common interface. Ultimately, vendors want your data on their systems – and that means making it easy to get the data from a competing vendor’s system; lock-in is not what vendors want. Talk to your vendor and ask for other users’ experiences to get confirmation of their claims.

Q. Based on factual information, where are you seeing the most common use cases for Object Storage?

A. There are many, and each vendor of cloud storage has particular markets. Backup is a common case, as are systems in the healthcare space that treat data such as scans and X-rays as objects.

Q. NAS filers only scale up not out. They are hard to manage at scale. Why use them anymore?

A. There are many NAS systems that scale out as well as up. NFSv4 support high degrees of scale out and there are file systems like Gluster that provide very large-scale solutions indeed, into the multi-petabyte range.

Q. Are there any specific uses cases to avoid when considering object storage?

A. Yes. Many legacy applications will not generate any savings or gains if moved to object storage.

Q. Would you agree with industry statements that 80% of all data written today will NEVER be accessed again; and that we just don’t know WHICH 20% will be read again?

A. Yes to the first part, and no to the second. Knowing which 80% is cold is the trick. The industry is developing smart ways of analyzing data to help with the issue of ensuring cached data is hot data, and that cold data is placed correctly first time around.

Q. Is there also the possibility to bring “compliance” in the object storage? (thinking about banking, medical and other sensible data that needs to be tracked, retention, etc…)

A. Yes. Many object storage vendors provide software to do this.

There has been a lot of buzz around cloud object storage recently. But before you get deep into all that cloud object storage can do, it’s good to take a step back and make sure you understand the basics. That’s what the SNIA Cloud Storage Initiative is planning to do on July 14^th at our live Webcast “Cloud Object Storage 101.”

Many organizations, like large service providers, have already begun to leverage software-defined object storage to support new application development and DevOps projects. Meanwhile, legacy enterprise companies are in the early stages of exploring the benefits of object storage for their particular business and are searching for how they can use cloud object storage to modernize their IT strategies, store and protect data, while dramatically reducing the costs associated with legacy storage sprawl.

This Webcast will highlight the market trends towards the adoption of object storage, the definition and benefits of object storage, and the use cases that are best suited to leverage an underlying object storage infrastructure.

Join us on July 14^th to learn:

• How to accelerate the transition from legacy storage to a cloud object architecture
• Understand the benefits of object storage
• Primary use cases
• How an object storage can enable your private, public or hybrid cloud strategy without compromising security, privacy or data governance

I hope you’ll register today to join my colleague, Nancy Bennis, Director of Alliances at Cleversafe (an IBM company), and me for this tutorial on cloud object storage.