A Software Based Fault Injection Framework for Storage Server

With increasing complexity of storage systems, the ability to gracefully handle errors at all layers of a storage server (array firmware, driver, file system, protocols) has become a key challenge to developers. This is crucial in scalable storage environment, where error handling has to be synchronized across multiple nodes. This makes software fault injection at various layers of the stack more important in storage development and testing. Currently there is no single infrastructure that allows selective injection of faults in a typical storage server implementation. While investigating this problem, we have studied available options and designed a framework that uses combination of Kprobe, frysk, at system and protocol layer, and custom firmware fault injection mechanism that can simulate transient and hard errors at various layers.

Learning Objectives

Typical Storage failures events
Simulating errors that cause Data Unavailability, Data Loss
Usage of open source tools in implementing fault injection framework in storage development
Robust and Fault tolerant storage stack design aspects