Netflix Studios produces petabytes of media content accounting for billions of media assets. These assets are managed, created, edited, encoded, and rendered by artists working on a multitude of workstation environments that run on cloud, from different parts of the globe. Artists working on a project may only need access to a subset of the assets from a large corpus. Artists may also want to work on their personal workspaces on intermediate content, and would like to keep only the final copy of their work persisted on cloud.

In file systems, large sequential writes are more beneficial than small random writes, and hence many storage systems implement a log structured file system. In the same way, the cloud favors large objects more than small objects. Cloud providers place throttling limits on PUTs and GETs, and so it takes significantly longer time to upload a bunch of small objects than a large object of the aggregate size. Moreover, there are per-PUT calls associated with uploading smaller objects. In Netflix, a lot of media assets and their relevant metadata is generated and pushed to cloud.

This presentation will outline the architectures for the top three platforms in each of these two categories, Von Neumann and Programmable Logic. Showing how vendors like NVIDIA, Pensando, Marvell, Achronix, Intel, and Xilinx have chosen to architect their solutions. We will then weigh the merits and benefits of each approach while also highlighting the performance bottlenecks. By the end of the presentation, it may be fairly clear where the industry is headed, and which solutions may eventually win out.

Storage interfaces have evolved more in the past 3 years than in the previous 20 years. In Linux, we see this happening at two different layers: (i) the user- / kernel-space I/O interface, where io_uring is bringing a low-weight, scalable I/O path; and (ii) and the host/device protocol interface, where key-values and zoned block devices are starting to emerge. Applications that want to leverage these new interfaces have to at least change their storage backends.

Cloud storage footprint is in exabytes and exponentially growing and companies pay billions of dollars to store and retrieve data. In this talk, we will cover some of the space and time optimizations, which have historically been applied to on-premise file storage, and how they would be applied to objects stored in Cloud. Deduplication and compression are techniques that have been traditionally used to reduce the amount of storage used by applications.

Microservices architectures enhance the ability for modern software teams to deliver applications at scale and have expanded the distributed nature of application. But as an application’s footprint grows, the challenge is to understand and control interactions among services within these environments. Enabling Service Mesh controls the communication, configuration and behavior of microservices in an application. A service mesh is a dedicated infrastructure layer for handling service-to-service communication in any microservice, public cloud or Kubernetes architecture.

Reducing the amount of data is a huge advantage of saving a total cost of ownership for a distributed storage system. To do this, a deduplication method which removes redundant data is being used as one of the promising solutions to save storage capacity. However, in practice, traditional deduplication methods designed for a local storage system is not suitable for a distributed storage system due to several challenging issues.

The QEMU emulated NVMe device is used by developers and users alike to develop, test and verify device drivers and tools. The emulated device is in rapid development and with QEMU 6.0, the device was updated to support a number of core additional features such as an update to NVMe v1.4, universal Deallocated and Unwritten Logical Block Error support, enhanced PMR and CMB support as well as a number of experimental features such as Zoned Namespaces, multipath I/O, namespace sharing and DIF/DIX end-to-end data protection.

Current enterprise storage devices have to service many diverse and continuously evolving application workloads (For e.g., OLTP, Big Data/Analytics and Virtualization). These workloads combined with additional enterprise storage services like deduplication, compression, snapshots, clones, replication, tiering etc. result in complex I/Os to the underlying storage. Traditional storage system tests make use of benchmarking tools, which generate a fixed and constant workload, comprised of a single or few I/O access patterns and are not sufficient for enterprise storage testing.

The de-facto way of copying data in I/O stack has been pulling it from one location followed by pushing to another. The farther the application, requiring copy, is from storage, the longer it takes for trip to be over. With copy-offload the trip gets shorter as the storage device presents an interface to do internal data-copying. This enables the host to optimize the pull-and-push method, freeing up the host CPU, RAM, and the fabric elements.