Cylon: Fast and Accurate Full-System Emulation of CXL-SSDs
San Tomas + Lawrence
Mon Sep 28 | 2:10pm
Abstract
CXL-SSDs promise to collapse the boundary between memory and storage by attaching SSDs behind CXL.mem, with a small DRAM cache serving sub-microsecond hits and NAND providing terabyte-scale capacity at tens-of-microseconds miss latency. Yet the tools to evaluate them have not kept up. Commercial prototypes such as Samsung's CMM-H are largely opaque, with firmware-controlled cache management and no policy knobs. FPGA platforms abstract away NAND behavior. Trace-driven and cycle-accurate simulators are orders of magnitude too slow to run real workloads. QEMU's upstream CXL support inflates every access to roughly 15 μs through MMIO and VM-exits, misrepresenting the sub-µs hit path that defines a CXL-SSD.
We present Cylon, to our knowledge, the first full-system emulator that combines speed, fidelity, and extensibility for CXL-SSD research. Built on FEMU, Cylon introduces a hybrid execution path: Dynamic EPT Remapping eliminates VM-exit overhead on cache hits so loads and stores complete with hit-path latency bounded by remote-NUMA access, while misses are trapped into FEMU for faithful NAND timing modeling. Shared EPT Memory removes the syscall and page-walk overhead of cache residency updates, reducing miss-path latency from 23 μs to 16 μs. On top of these mechanisms, Cylon provides a pluggable caching framework, including FIFO, LIFO, CLOCK, S3FIFO eviction, and next-line prefetching, and an application-level hint API for hardware-software co-design.
We validate Cylon against a Samsung CMM-H prototype across microbenchmarks, Redis, and the GAP Benchmark Suite. Cylon captures the qualitative bimodal latency behavior and bandwidth-cliff characteristics of real hardware, runs unmodified operating systems and applications at near bare-metal speed, and provides an idealized baseline that isolates cache-policy effects from prototype-specific artifacts.
The talk will cover (1) why faithful CXL-SSD emulation is harder than it looks, the latency floor QEMU-CXL hits and why; (2) the KVM/EPT mechanisms behind Cylon's hybrid path; (3) what eviction and prefetching policy sweeps reveal about cache management under realistic workloads; and (4) the design space Cylon opens up beyond CMM-H for next-generation CXL-storage architectures.
Related Sessions
Introduction to STT-MRAM
Spin-Transfer Torque MRAM (STT-MRAM) has become a compelling and widely deployed non-volatile memory technology across a growing range of applications. This talk provides a technical introduction to how STT-MRAM works, including its underlying physics and device operation, and why it offers unique advantages compared to conventional memory technologies. We will explore four key application domains where STT-MRAM is already making an impact and where it is poised to grow in the near future, spanning standalone and embedded products. Particular attention will be given to STT-MRAM’s intrinsic robustness, including its significantly higher immunity to magnetic fields—more than an order of magnitude greater than hard disk drives. The session will conclude with a perspective on the current state of MRAM deployment and the roadmap ahead, highlighting ongoing developments and opportunities for adoption in next-generation AI systems.
Unordered I/O (UIO) Enablement for CXL P2P Data Paths in AI Systems
CXL is emerging as a key fabric for disaggregated memory and accelerator interconnects in AI systems, where efficient peer-to-peer (P2P) data movement and bandwidth aggregation would be beneficial. While CXL inherits PCIe transaction semantics, emerging capabilities such as PCIe Unordered I/O (UIO) might require explicit software management to be made really useful.
This work presents UIO-driven P2P enablement in the Linux kernel for CXL-based systems, extending PCI and CXL subsystems to support device-to-device data paths aligned with relaxed ordering semantics. We discuss the modifications for propagation of transaction attributes, and integration with P2PDMA, enabling correct and efficient utilization of hardware-supported unordered transactions. We discuss the key differences and changes to be done for UIO capable cxl.mem and cxl.io transactions, which makes CXL a viable fabrics for AI systems.
To support early-stage development, we incorporate a UIO-capable CXL model in QEMU, enabling functional validation of ordering behavior and topology-aware routing in the absence of complete hardware platforms.
Together, these contributions provide a practical path to prototype and evaluate UIO-enabled data movement in CXL fabrics. This enables scalable communication patterns important for AI workloads such as tensor movement and KV-cache sharing, and establishes a foundation for future hardware realization and upstream kernel support.
Cylon: Fast and Accurate Full-System Emulation of CXL-SSDs
CXL-SSDs promise to collapse the boundary between memory and storage by attaching SSDs behind CXL.mem, with a small DRAM cache serving sub-microsecond hits and NAND providing terabyte-scale capacity at tens-of-microseconds miss latency. Yet the tools to evaluate them have not kept up. Commercial prototypes such as Samsung's CMM-H are largely opaque, with firmware-controlled cache management and no policy knobs. FPGA platforms abstract away NAND behavior. Trace-driven and cycle-accurate simulators are orders of magnitude too slow to run real workloads. QEMU's upstream CXL support inflates every access to roughly 15 μs through MMIO and VM-exits, misrepresenting the sub-µs hit path that defines a CXL-SSD.
We present Cylon, to our knowledge, the first full-system emulator that combines speed, fidelity, and extensibility for CXL-SSD research. Built on FEMU, Cylon introduces a hybrid execution path: Dynamic EPT Remapping eliminates VM-exit overhead on cache hits so loads and stores complete with hit-path latency bounded by remote-NUMA access, while misses are trapped into FEMU for faithful NAND timing modeling. Shared EPT Memory removes the syscall and page-walk overhead of cache residency updates, reducing miss-path latency from 23 μs to 16 μs. On top of these mechanisms, Cylon provides a pluggable caching framework, including FIFO, LIFO, CLOCK, S3FIFO eviction, and next-line prefetching, and an application-level hint API for hardware-software co-design.
We validate Cylon against a Samsung CMM-H prototype across microbenchmarks, Redis, and the GAP Benchmark Suite. Cylon captures the qualitative bimodal latency behavior and bandwidth-cliff characteristics of real hardware, runs unmodified operating systems and applications at near bare-metal speed, and provides an idealized baseline that isolates cache-policy effects from prototype-specific artifacts.
The talk will cover (1) why faithful CXL-SSD emulation is harder than it looks, the latency floor QEMU-CXL hits and why; (2) the KVM/EPT mechanisms behind Cylon's hybrid path; (3) what eviction and prefetching policy sweeps reveal about cache management under realistic workloads; and (4) the design space Cylon opens up beyond CMM-H for next-generation CXL-storage architectures.
Enabling Terabytes of Byte-Addressable Memory using CXL Hybrid Memory Devices
The latest versions of the CXL standard allow system architects to consider future systems using secure disaggregated and distributed shared memory. With the immediate concerns of increasing DRAM costs, system designers are preparing to deploy CXL devices that reuse already-purchased DRAM DIMMs in servers supporting newer versions of DDR, expanding system memory capacity without incurring additional DRAM costs. This presentation explores another means to reduce the cost of memory using CXL: a CXL Type 3 device variant, sometimes referred to as CXL-SSD or CMM-H, which combines DRAM and NAND storage yet offers a host byte-addressable access to tens of TBs of memory.
For a CXL hybrid media device with DRAM and SSD, the CXL device designer needs to maintain an internal balance between the CXL device’s PCIe interface, internal DDR bandwidth, and SSD page size. The CXL device uses the DRAM to cache SSD pages and mitigate SSD latency for host CPU CXL.mem requests, but the CXL device still maintains the contract between the host physical address (HPA) and device physical address (DPA) regardless of where the SSD page resides. CXL devices with hybrid memory require a host CPU and its operating system to configure host physical address spaces an order of magnitude larger than current systems support, and designers need to find new means to reduce the metadata needed to support such large physical address spaces.
Numerous workloads benefit from lower-overhead access to byte-addressable memory over IO, but some memory latency-sensitive applications require techniques to reduce access latency to data stored on a CXL device. Some CXL host CPUs provide hardware support to mitigate memory latency to CXL. For a CXL DRAM-SSD hybrid media device, an application could provide information to the CXL device—for example, receiving notifications to pre-stage data in the CXL device’s DRAM cache before the application accesses the DPAs. An application that provides pre-staging hints or access pattern hints to a CXL hybrid media device allows the device to use its DRAM for caching SSD pages more efficiently.
Though CXL hybrid memory devices may have microsecond latencies upon missing in the DRAM cache, in the future, with the rise of systems using scale-up Ethernet, higher latency memory access may not be unique to CXL hybrid devices or CXL memory pools. Past and current academic research has evaluated different use cases and solutions for host CPUs to tolerate longer-latency memory accesses.
This presentation uses design choices and workload characterization data gathered from XCENA’s CXL 3.x device, MX1—the first ASIC implementation of a CXL 3.x hybrid media CXL device—to illustrate key points.
MRAM for Data Centers in Space
Today, many organizations are actively developing strategies for orbital space data centers. A central challenge is identifying storage and memory technologies capable of meeting the unique constraints of the space environment. MRAM inherently satisfies these requirements, offering radiation tolerance and robust operation across wide temperature ranges. As these characteristics are intrinsic to MRAM technology, it represents a compelling and natural memory solution for the next generation of data centers deployed in orbit.
MRAM for AI
In recent years, the computational power of AI chips has grown much faster than memory bandwidth and data transfer efficiency. Data centers may soon be choked by the demand for high performance data access and the cost of providing it. Jump in MRAM! With the potential to allow AI inference models to be built into low energy devices at the edge instead of using expensive high performance data center options, MRAM can move the goalposts.
