Testing a Distributed File System

webinar

Author(s)/Presenter(s):

Mukul Kumar Singh

Cloudera

Library Content Type

Presentation

Library Release Date

Focus Areas

Storage Management

Abstract

Filesystem is an exciting software system to develop, because there is lots of complexity involved in implementing a new filesystem. Because of the complex nature of the filesystem, testing it is also an equally difficult task as well. A distributed file system spanning over hundred to thousand of nodes over it adds to the complexity of both development and testing of the filesystem. In this talk we will talk about how we tested Ozone, which is a distributed file system. Ozone is composed of metadata nodes like Ozone Manager and Storage Container Manager and the datanodes where the actual data is stored. In this talk we will discuss various parameters on which a filesystem should be tested, aka functional, non-functional, performance, chaos, scalability, integration and longevity testing. We will share our experience of developing and then testing a distributed file system and the learnings from doing this analysis. We will also try to share the scope and timelines of when some of these testing was done in Ozone and how this helped in evolving Ozone to a mature filesystem.