Over the past 30+ years, I/O workflows have evolved and changed dramatically. Data access via file protocols has morphed to Data access via object protocols. Permission granularity control requirements have grown from modebits to Access Control Lists to Identity Policies. The sheer size of the datasets both in capacity and number of items has grown exponentially. Modern workflows, including those related to AI and Machine learning, involve access to data across multiple protocols with vastly different methods of permissioning and control. Building a bridge between file and object protocols in such a way to ensure the SAME access rights to SAME dataset regardless of protocol is challenging. We believe that this can be solved by a protocol-agnostic approach where, internally, there are no NFS (or SMB) files, no S3 objects, no NTFS ACLs, and no NFS modebits. Instead, the permissioning model is flexible and provides consistent access for a user, regardless of protocol used, such that a user can create via NFS, analyze via SMB, and share via S3 without having to move, relocate, or re-permission data.
Outline and understand the challenges of at scale multi-protocol data sharing over NFS, SMB and object storage (S3).
Discuss the VAST approach of managing users, metadata, permissions and security policies.
Performance optimization for maintaining consistent access rights across protocols while allowing for flexibility depending on use case.