High-Throughput Cloud Storage Over Faulty Networks

Storage systems increasingly rely on the Internet as the medium of data transfer. The Internet, as a high-bandwidth, high-latency, high-packet loss connection is very different from the clean networks of typical SANs. Under such conditions, TCP’s capabilities are often stretched to their breaking point. In this presentation we detail the methods we used to overcome random network slowdowns, packet drops, congestion control, and other challenges. Our result: we achieved storage throughputs on the Internet that were 80% that of the same test on a low-latency, zero packet loss LAN.

Learning Objectives

Why the network layer is increasingly important in today's storage systems, as cloud storage takes off, and the NIC and Internet speeds begin to eclipse the speed of a local hard drive.
Features and limitations of TCP, including congestion control, window scaling, error handling, order preservation and how they pertain to modern cloud based storage systems.
How to achieve reliable low-latency delivery of messages over networks with unpredictable reliability and performance, in ways that are easy to implement, manageable and widely supported.