Deployment Planning and Optimization for Big Data & Cloud Storage Systems

With the rise of big data analytics systems, IT spending on storage system is increasing. In order to minimize costs architects must optimize system capacities and characteristics. Current capacity planning is mostly based on trial and errors as well as rough resource estimations. With increasing hardware diversity and software stack complexity this approach is not efficient enough. This session presents a novel modeling framework, built with Intel® CoFluent™ Studio, that can be used before system provisioning for cluster capacity planning, performance evaluation and optimization. The methodology uses a top-down approach to model behavior of a complete software stack and simulates the activities of cluster components including storage, network and processors. In addition, simulations can scale to a large number of server nodes while attaining good accuracy and fast simulation speeds (even faster than native execution).

Learning Objectives

Storage system modeling technology for Big Data & Cloud
HDFS and Swift simulation vs. measurements
Real uses case: the planning and optimization of a video streaming cluster