Compacting Smaller Objects in Cloud for Higher Yield

Library Content Type:
Publish Date: 
Wednesday, September 29, 2021
Event Name: 
Event Track:

In file systems, large sequential writes are more beneficial than small random writes, and hence many storage systems implement a log structured file system. In the same way, the cloud favors large objects more than small objects. Cloud providers place throttling limits on PUTs and GETs, and so it takes significantly longer time to upload a bunch of small objects than a large object of the aggregate size. Moreover, there are per-PUT calls associated with uploading smaller objects. In Netflix, a lot of media assets and their relevant metadata is generated and pushed to cloud. Most of these files are between 10s of bytes to 10s of kilobytes and are saved as small objects on Cloud. In this talk, we would like to propose a strategy to compact these small objects into larger blobs before uploading them to Cloud. We will discuss the policies to select relevant smaller objects, and how to manage the indexing of these objects within the blob. We will also discuss how different cloud storage operations such as reads and deletes would be implemented for such objects. Finally, we would showcase the potential impact of such a strategy on Netflix assets in terms of cost and performance.

  • Learn about per request throttle & costs in cloud storage
  • Leveraging machine learning techniques to select relevant objects for compaction
  • Cloud operations such as reads, writes, deletes, updates on compacted objects
  • Impact of the proposal on Netflix assets

Watch video: