How to build a reliable, scalable, parallel filesystem solution using Cloud

In the last couple of years, the case for HPC in the cloud is growing stronger. But still, the HPC industry lies far behind enterprise IT in its willingness to outsource computational power. One of its reason being storage - as none of the built-in storage solutions available across the public cloud providers are suitable for applications with high bandwidth requirements. A parallel, clustered file system built on top of block storage (Ex: AWS EBS) forms a good solution. However, there exist multiple architectural approaches (such as when & why to use placement groups, autoscaling, auto recovery, replication across availability zones etc.) one can take to build parallel, clustered filesystem. Apart from building, there exists a need for effectively managing the cloud resources used for building the filesystem as it could quickly become expensive. The proposed presentation is aimed to discuss in detail different architectures (weighing Pro’s & Con’s) that can be used to build a reliable parallel filesystem in the cloud (showcasing AWS, IBM Spectrum Scale as an example) and data lifecycle techniques that help reduce OPEX cost by effectively managing parallel filesystem in the cloud.