Taming the Flood: Massively Scalable Semi-P2P Content Distribution

Sometimes demand for popular data exceeds the capability of storage servers to deliver it, but new protocols offer a solution. Protocols that leverage client resources can scale capacity to meet any level of demand--a famous example being Bittorrent. Yet there are many challenges with dynamically creating, tracking and seeding torrents to satisfy millions of users, accessing petabytes of data, across an enterprise-class storage system. Problems compound when peers are untrusted, potentially malicious and sometimes uncooperative. In this presentation we consider whether the problem can be solved for the general case, and evaluate the benefits of predictive caching, machine learning, and modifications to the Bittorrent protocol to create a storage system of truly unlimited capacity for content distribution.

Learning Objectives

Massively scalable enterprise-class content distribution using torrent-based protocols
Intelligent, predictive server-side caching based on past behavior and current state of clients
How client trust models affect the efficiency of a P2P file distribution scheme