Erasure Coding in Windows Azure Storage

Erasure coding is being adopted in planet-scale cloud storage systems, from HDFS in Facebook, GFS II in Google to Windows Azure Storage in Microsoft. It promises to reduce hardware and operational costs by 50%. While many systems settle with conventional erasure coding schemes, invented more than 40 years ago and primed in deep space communication, Microsoft Research and Windows Azure Storage have partnered and designed advanced new erasure coding schemes - optimized specifically for cloud storage - to achieve significantly more savings. This presentation is targeted at storage developers with strong desire to reduce storage system costs and operate more economically. We will first review conventional erasure coding schemes and then explain our advanced new erasure codes. We will describe how the new codes are being used in Windows Azure Storage and share lessons/experience learned from real production.

Learning Objectives

Refresh my knowledge of erasure coding. (Some knowledge is assumed – this is *not* a tutorial on Erasure Coding. For basics, see Jim Plank’s USENIX-FAST tutorial “Erasure Codes for Storage Applications”.)
Get an overview of erasure coding as currently applied to large-scale storage systems, specifically Cloud Storage.
Understand a class of erasure codes (used in Windows Azure Storage) that is useful and economical for Cloud Storage systems.