Architecting An Enterprise Storage Platform Using Object Stores

Author(s)/Presenter(s):
Library Content Type:
Publish Date: 
Monday, September 16, 2013
Event Name: 
Focus Areas:
Abstract: 

While object storage systems such as S3 and Swift are exhibiting rapid growth, there is still an impedance mismatch between their feature set and enterprise requirements. This talk dives into the design and architecture of MagFS: a strongly consistent and multi-platform distributed file system that layers itself on top of multiple object storage systems. In particular, it covers the challenges of using eventually consistent object stores, optimizing both data and metadata traffic for wide-area network communication and mobile devices, and how MagFS delivers an on-premises security model while still being able to leverage off-premises storage. This talk also discusses how specific enterprise requirements have influenced the technical design of MagFS and some of the surprises we encountered during our design and implementation.