Balancing Storage Utilization Across a Global Namespace

Global namespaces represent the pinnacle of scalability as no central authority need be consulted to locate or update a resource. Just as DNS has enabled the Internet to scale to billions of hosts, global namespaces have much utility for scaling storage systems to Petabytes and beyond. Yet there are trade-offs to be made. The less dynamic the namespace the greater the scalability, but a more rigid namespace restricts data migration and rebalancing choices. We describe the trade-offs we made in designing a namespace that scales to Exabytes and how we deal with storage imbalance and expansion.

Learning Objectives

What a global namespace is, and the benefits they provide over traditional metadata or lookup services.
Limitations imposed by a rigid namespace, in terms of where data can be migrated or moved to without causing the namespace mapping to change or expand in size.
The design of our global namespace and algorithms employed to balance utilization across a storage system of thousands of nodes.