Real-Time Networking with Stanford’s HOMA Protocol
Abstract
At SDC 2023 and SDC 2024 we presented Homa, a reliable datacenter transport-layer protocol invented by John Ousterhout at Stanford University. We were able to demonstrate that Homa and TCP can co-exist in the same network “peacefully”, and how Homa can be accelerated by something similar to what Offload Engines do for TCP.
Today, (almost) everybody’s eyes are on datacenter infrastructure and AI clusters because of the challenging Terabit line rates and the high volume numbers of network ports. Nevertheless, there are many other applications which can also benefit from Homa:
So-called systems-of-systems such as zone-based automotive networks, telecommunication core networks, converged IT/OT networks in industrial automation, backbones in humanoid robots, or Sensor Open Systems Architectures in aerospace & defense need a reliable transport-layer protocol which minimizes tail latency and optimizes infrastructure efficiency to deliver bandwidths of 10 Gbps, and more.
Many of these systems require hard real-time behavior which - in theory - matches with Homa’s attributes of being a) message-based, b) connection-less, c) using receiver-driven congestion control, and d) run-to-completion (SRPT). Time Sensitive Networking (TSN) has become the de-facto choice using IEEE standard Ethernet. TSN comes with time-synchronization, traffic shaping, reliability and resource management and works well with higher-layer protocols such as TCP.
Hence, in our presentation this year we will share insights on how Homa can be run over TSN and, in particular, the effects of different forms of traffic shaping on Homa’s core functions. We plan to start with a refresher on Homa and then dive into architecture choices when running TSN and multiple Gbps. This is complemented by presenting “Light Rabbit” which is a cost-optimized variant of CERN's White Rabbit high-accuracy time synchronization which utilizes programmability of modern PLLs for frequency and phase synchronization to achieve nano-second accuracy. We close by presenting first experimental results of running Homa over TSN to achieve the best of both worlds.
Related Sessions
Why S3FS Fails in AI/ML — and How to Achieve Scalable POSIX Access Anyway
Mounting S3-compatible storage via S3FS seems like an easy way to enable POSIX-like access in Kubernetes. But in real AI/ML workloads—e.g., training with PyTorch or TensorFlow—we hit major issues: crashes from incomplete writes, vanished checkpoints, inconsistent metadata, and unpredictable I/O latency.
In this session, we’ll share how we overcame these challenges by designing a scalable, POSIX-compliant distributed file system that still leverages the cost-effectiveness of object storage. Instead of abandoning object storage, we rebuilt the access layer for better consistency, performance, and observability in large-scale environments.
Attendees will gain insight into architectural trade-offs, POSIX compliance in user space, Kubernetes integration via CSI and Operators, and observability benchmarks collected from real production AI training clusters.
Ideal for platform engineers, MLOps, and K8s architects seeking reliable, scalable storage for data-heavy workloads.
This is an intermediate session; attendees should be comfortable with object storage, file storage, and the basic concepts of the Kubernetes CSI driver.
Highly Scalable, Masterless, Distributed Filesystem at Rubrik
Rubrik is a cybersecurity company protecting mission critical data for thousands of customers across the globe including banks, hospitals, and government agencies. SDFS is the filesystem that powers the data path and makes this possible. In this talk, we will discuss challenges in building a masterless distributed filesystem with support for data resilience, strong data integrity, and high performance which can run across a wide spectrum of hardware configurations including cloud platforms. We will discuss the high level architecture of our FUSE based filesystem, how we leverage erasure coding for maintaining data resilience and checksum schemes for maintaining strong data integrity with high performance. We will also cover the challenges in continuously monitoring and maintaining the health of the filesystem in terms of data resilience, data integrity and load balance. Further we will go over how we expand and shrink resources online from the filesystem. We will also discuss the need and challenge of providing priority natively in our filesystem to support a variety of workloads and background operations with varying SLA requirements. Finally, we will also touch on the benefits and challenges of supporting encryption, compression, and de-duplication natively in the filesystem.
Scalable Metadata in Distributed File Systems: Revisiting the GoogleFS Design for Exabyte-Scale Namespaces
GoogleFS introduced the architectural separation of metadata and data, but its reliance on a single active master imposed fundamental limitations on scalability, redundancy, and availability. This talk presents a modern metadata architecture, exemplified by SaunaFS, that eliminates the single-leader model by distributing metadata across multiple concurrent, multi-threaded servers. Metadata is stored in a sharded, ACID-compliant transactional database (e.g., FoundationDB), enabling horizontal scalability, fault tolerance through redundant metadata replicas, reduced memory footprint, and consistent performance under load. The result is a distributed file system architecture capable of exabyte-scale operation in a single namespace while preserving POSIX semantics and supporting workloads with billions of small files.
Open Flash Platform: An Initiative for Open, Highly Efficient AI Storage
Enterprise IT infrastructures face soaring AI and analytics demands, driving the need for storage that leverages existing networks, cuts power-hungry server counts, and frees CAPEX for AI. Yet current solutions create isolated silos: proprietary, server-based systems that waste power, lack cloud connectivity, and force large teams to manage multiple silo technologies—locking data behind vendor walls and hampering AI goals. Modeled on the Open Compute Project, the Open Flash Platform (OFP) liberates high-capacity flash through an open architecture built on standard pNFS which is included in every Linux distribution. Each OFP unit contains a DPU-based Linux instance and network port, so it connects directly as a peer—no additional servers. By removing surplus hardware and proprietary software, OFP lets enterprises use dense flash efficiently, halving TCO and increasing storage density 10×. Early configurations deliver up to 48 PB in 2U and scale to 1 EB per rack, yielding a 10× reduction in rack space, power, and OPEX and a 33 % longer service life. This session explains the vision and engineering that make OFP possible, showing how open, standards-based architecture can simplify, scale, and free enterprise data.
Choosing Your AI Storage Protocol: A Deep Dive into SMB and NFS Performance, Tuning, and Overhead
The performance of network file protocols is a critical factor in the efficiency of the AI and Machine Learning pipeline. This presentation provides a detailed comparative analysis of the two leading protocols, Server Message Block (SMB) and Network File System (NFS), specifically for demanding AI workloads. We evaluate the advanced capabilities of both protocols, comparing SMB3 with SMB Direct and Multichannel against NFS with RDMA and multistream TCP configurations. The industry-standard MLPerf Storage benchmark is used to simulate realistic AI data access patterns, providing a robust foundation for our comparison. The core of this research focuses on quantifying the performance differences and identifying the operational and configuration overhead associated with each technology.
New Transports in Samba: QUIC and SMB-Direct Support
The Samba file server is evolving beyond traditional TCP-based transport. This talk introduces the latest advancements in Samba's networking stack, including full support for SMB over QUIC, offering secure, firewall-friendly file sharing using modern internet protocols. We’ll also explore the ongoing development of SMB over SMB-Direct (RDMA), aimed at delivering low-latency, high-throughput file access for data center and high-performance environments. Join us for a deep dive into these transport innovations, their architecture, current status, and what's next for Samba’s high-performance networking roadmap.
