Ozone - Architecture and Performance at billions’ scale

Library Content Type:
Publish Date: 
Wednesday, September 29, 2021
Event Name: 
Event Track:
Abstract: 

Object stores are known for ease of use and massive scalability. Unlike other storage solutions like file systems and block stores, object stores are capable of handling data growth without increase in complexity or developer intervention. Apache Hadoop Ozone is a highly scalable Object Store which extends the design principles of HDFS while maintaining a 10-100x scale compared to HDFS. It can store billions of keys and hundreds of petabytes of data. With the massive scale there is a requirement for it to have very high throughput while maintaining low latency. This talk discusses the Ozone architecture and design decisions which were significant for achieving high throughput and low latency. It will detail how Ozone supports metadata and data layer without compromising throughput or latency. Further the talk discusses the hardships and challenges related to resource management and good performance in Ozone. It would cover some major pain points and present the performance issues in broad categories.

  • How Ozone architecture helps in achieving massive scale
  • How Ozone extends the design principles of HDFS
  • Resource management for data and metadata layer
  • Overview of performance issues

Watch video: