Solving the Framework-Storage Gap in Big Data

As datasets continue to grow, storage has increasingly become the critical bottleneck for enterprises leveraging Big Data frameworks like Spark, MapReduce, Flink, etc. The frameworks themselves are driving much of the exciting innovation in Big Data, but the complexity of the underlying storage systems is slowing the pace that data assets can be leveraged by these frameworks. Traditional storage architectures are inadequate for distributed computing and the size of today’s datasets.

In this talk, Haoyuan Li, co-creator of Tachyon (and a founding committer of Spark) and CEO of Tachyon Nexus will explain how the next wave of innovation in storage will be driven by separating the functional layer from the persistent storage layer, and how memory-centric architecture through Tachyon is making this possible. Li will describe the future of distributed file storage and highlight how Tachyon supports specific use cases.

Li will share the vision of the Tachyon project, highlight exciting new capabilities, and give a preview at upcoming new features. The project is one of the fastest growing big data open source projects. It is deployed at many companies, including Alibaba, Baidu and Barclays. Tachyon manages hundreds of machines in some of the production deployments and brings orders of magnitude improvement. In addition, Tachyon has attracted more than 200 contributors from over 50 institutions, including Alibaba, Redhat, Baidu, Intel, and IBM.