Object Storage Technology

Object Storage is a generalized data container with uses in cloud storage, HPC file systems, and custom applications that roll their own indexing and metadata layers over objects. This tutorial provides a survey of these different kinds of objects, their APIs, and the applications that use them. The OSD (Object Storage Device) command set for SCSI provides a secure, general purpose mechanism used in HPC file systems via the NFSv4.1 pNFS (parallel NFS) protocol. The Amazon S3 object interface uses a web-based transport to provide cloud-based storage, and there are a number of similar interfaces in the open source community such as Swift and Eucalyptus. The differentiating features of object storage systems include the access protocols (SCSI, RPC, REST), performance (high-speed LAN or WAN), security mechanisms, replication and reliability schemes, metadata and indexing services. As a result, different application domains have evolved their own Object Storage ecosystems.

Learning Objectives

Understand the OSD and NFSv4.1 pNFS standards
Understand web objects and their “REST-ful” APIs
Understand the tradeoffs between POSIX file system semantics and cloud storage semantics