Emerging Deep Learning/ Machine learning and cloud native Applications at data center scale demand terabytes of data flowing across the storage/ memory hierarchy, straining interconnect bandwidth and component capacities. The Industry has responded with a wide range of solutions like process node shrink, higher capacity devices, new tiers, innovative form factors, new interconnect technologies and fabrics, new types of compute architectures, new algorithms and more to creatively leverage storage/memory tiering.

Main memory dominates data center server cost, and hence, data center operators are exploring alternative technologies such as CXL-attached memory to improve cost without jeopardizing performance. Introducing multiple tiers of memory introduces new challenges, such as selecting the appropriate memory configuration for a given workload mix. In particular, we observe that inefficient configurations increase cost by up to 2.6× for clients, and resource stranding increases cost by 2.2× for cloud operators.