Cutting the Cord: Why We Took the File System Out of Our Storage Nodes

A major component of request latency is the time back-end servers spend reading and writing data. Object storage systems commonly use JBOD servers together with a general purpose file system to store object data in files. Yet there are far more efficient ways of implementing the back-end storage. In this presentation, we explore the lessons we learned, problems we encountered, and resulting performance gains in the process of creating a new back-end storage format. This format is applied to the raw disk over the SCSI interface, is append-only in its operation (SMR and SSD friendly) and enables single-seek access to any object on the disk. The result is significantly increased IOPS together with significantly reduced latency.

Learning Objectives

General purpose filesystems are not efficient for large scale object storage
An example of a raw-disk data format
Append-only writes work efficiently for both SMR and PMR drives
Memory requirements of a highly performant raw-disk storage backend