Storage Developer Conference Abstracts

Break Out Sessions and Agenda Tracks Include:

Note: This agenda is a work in progress. Check back for updates on additional sessions as well as the agenda schedule.

Cold STORAGE Data

Introducing the View of SNIA Japan Cold Storage Technical Working Group on "Cold Storage"

Kazuhiko Kawamura, Senior Manager - System Engineering, Sony Corporation

Abstract

Introducing the view of SNIA Japan Cold Storage Technical Working Group on "Cold Storage"

Nowadays, huge amount of data is being generated everyday regardless of our awareness. Since nobody knows how much value the data would have in the future, we are obliged to retain more and more data, a.k.a. "cold data" at a certain cost.

Fortunately, we have quite a few options, however, there is apparently no standard yardstick to make our decision. Therefore, we have been discussing this topic for many months. Today, I'd like to introduce our current view on the cold storage with a bit of taxonomy by various characteristics of such storage devices.

Distributed Systems

Modern Erasure Codes for Distributed Storage Systems

Srinivasan Narayanamurthy, Member Technical Staff, NetApp

Abstract

Traditional disk based storage arrays commonly use various forms of RAID for failure protection and/or performance improvement. Modern distributed storage systems are built as shared nothing architectures using commodity hardware components and have different failure characteristics.

RAID has changed very little ever since its introduction three decades ago. There are several new and powerful constructions of erasure codes in the recent past (post- 2007) that were handcrafted to meet the needs of modern storage architectures. Several notable ones are fountain codes, locally repairable codes (LRC) and regenerating codes. However, these codes have made very little impact in the industry, because of the highly resistant nature of the storage vendors to investigate this space.

The subject of erasure codes is core to storage technologies, but is sadly ignored by most storage conventions. One of the reasons could be the complexity involved in understanding the space.

This talk will provide a non-mathematical, non-marketing, no-nonsense spin to the space of modern erasure codes. By the end of this talk, the audience will be able to pick specific categories from a bouquet of erasure codes that best suit a given storage system design.

Learning Objectives

  • History and classification of erasure codes.
  • Technical deep-dive into modern erasure codes.
  • Storage system parameters’ trade-off.
  • Storage system design & suitable codes.
  • Who is doing what in this space?

Erasure Coding

SNIA Tutorial:
Optimize Storage Efficiency & Performance with Erasure Coding Hardware Offload

Dror Goldenberg, VP of Software Architecture, Mellanox Technologies

Abstract

Nearly all object storage, including Ceph and Swift, support erasure coding because it is a more efficient data protection method than simple replication or traditional RAID. However, erasure coding is very CPU intensive and typically slows down storage performance significantly. Now Ethernet network cards are available that offload erasure coding calculations to hardware for both writing and reconstructing data. This offload technology has the potential to change the storage market by allowing customers to deploy more efficient storage without sacrificing performance. Attend this presentation to learn how erasure coding hardware offloads work and how they can integrate with products such as Ceph.

Learning Objectives

  • Learn the benefits and costs of erasure coding
  • Understand how erasure coding works in products such as Ceph
  • See how erasure coding hardware offloads accelerate storage performance

/etc

Early Developer Experiences with New Windows Nano Server as Embedded Enterprise Storage Appliance Platform OS

Terry Spear, President & CTO, IOLogix, Inc.

Abstract

This presentation will cover early developer experiences using the new Windows 2016 Nano Server as an embedded platform OS for Enterprise Storage Appliances. We will briefly review available built-in storage technologies of this OS configuration. Experiences with the different development and test methodologies required for this new OS will be discussed. Finally, a specific storage application development case of extending Nano Server by adding a Fibre Channel SCSI Target will be covered in detail including preliminary performance results.


Hyperscaler Storage

Mark Carlson, Principal Engineer, Industry Standards, Toshiba

Abstract

Hyperscaler companies as well as large enterprises who build their own datacenters have specific requirements for new features in storage drives. The SNIA Technical Council has created a white paper on these requirements and how current and future standards and open source projects can address them.

This talk will present the results of the the TC research in this area and discuss how SNIA and other standards bodies are making changes to accommodate them.

Learning Objectives

  • Learn about Datacenter customer drive requirements
  • Learn about existing and new standards for drive interfaces
  • Learn about open source projects that address these requirement

File Systems

MarFS: Near-POSIX Access to Object-Storage

Jeff Inman, Software Developer, Los Alamos National Laboratory
Gary Grider, HPC Div Director, LANL

Abstract

Many computing sites need long-term retention of mostly-cold data, often referred to as “data lakes”. The main function of this storage-tier is capacity, but non-trivial bandwidth/access requirements may also exist. For many years, tape was the most economical solution. However, data sets have grown larger more quickly than tape bandwidth has improved, such that disk is now becoming more economically feasible for this storage-tier. MarFS is a Near-POSIX File System, storing metadata across multiple POSIX file systems for scalable parallel access, while storing data in industry-standard erasure-protected cloud-style object stores for space-efficient reliability and massive parallelism. Our presentation will include: Cost-modeling of disk versus tape for campaign storage, Challenges of presenting object-storage through POSIX file semantics, Scalable Parallel metadata operations and bandwidth, Scaling metadata structures to handle trillions of files with billions of files/directory, Alternative technologies for data/metadata, and Structure of the MarFS solution.


Data Integrity support for Silent Data Corruption in Gfarm File System

Osamu Tatebe, Professor, University of Tsukuba

Abstract

Data stored in storage are often corrupted silently without any explicit error. To cope with the silent data corruption, the file system level detection is effective; Btrfs and ZFS have a mechanism to detect it by adding checksum in each block. However, data replication is required to correct the corrupted data, which waste storage capacity in local file system. Gfarm file system is an open-source distributed file system, which federates storages even among several institutions in wide area. It supports file replicas to improve access performance and also fault tolerance. The number and the locations of file replicas are specified by an extended attribute of a directory or a file. This presentation describes a design and implementation of data integrity support for silent data corruption in Gfarm file system. Due to native and required feature of file replicas in Gfarm file system, the silent data corruption can be detected efficiently and the corrupted data can be corrected without any waste storage capacity.

Green Storage

Using SPEC SFS with the SNIA Emerald Program for EPA Energy Star Data Center Storage Program

Vernon Miller, Senior Software Engineer, IBM
Nick Principe, Principal Software Engineer, EMC

Abstract

The next storage platform category to be added into the EPA Data Center Storage program is File storage. Come learn what it takes in setting up a SNIA Emerald File testing environment with the SPEC SFS tools, the additional energy related instrumentation and data collection tools. Don't wait to be kicked in the "NAS" when an Energy Star rating gates selling your File storage solutions.

Learning Objectives

  • Understand how the EPA Energy Star Data Center Storage Program applies to file and NAS environments.
  • Understand the methodology used in the SNIA Emerald Program to evaluate energy consumption of file and NAS environments.
  • Understand how the SNIA Emerald Program uses SPEC SFS2014 to drive file and NAS workloads.

Hardware

Characterizing the Evolution of Disk Attributes via Absorbing Markov Chains

Rachel Traylor, Research Scientist, EMC

Abstract

The life cycle of any physical device is a series of transitions from an initial healthy state to ultimate failure. Since Markov chains are a general method for modeling state transitions, they can effectively model the transitions as well as life expectancy. As a specific example, HDD life cycle can be analyzed by developing a Markov model from various disk indicators (e.g. medium errors, RAS, usage). An illustration is given wherein the evolution of medium errors in HDDs is modeled by an absorbing Markov chain. Examples of methods to employ the information contained in the Markov chain are provided.

Learning Objectives

  • understand how a Markov chain may be used to model a variety of disk qualities/errors
  • understand the types of business/operations related answer such a model provides
  • gain insight into how one can employ such information to develop flexible business rules

The Curse of Areal Density

Ralph Weber, Consultant, ENDL Texas

Abstract

During the past half decade, the uninterrupted march of higher and higher areal densities has begun to encounter the unforgiving laws of physics. Where once the differences between the energies required write and read had not mattered one iota, the higher forces demanded by writes are putting pressure on a wide variety of I/O parameters that previously were thought to be inviolate for random access read/write storage devices. A new form of Whack A Mole has emerged in which point solutions for various problems are popping up faster than file systems designers can respond. Never mind that the real hobgoblin is licking his chops and preparing to claim more victims.

Learning Objectives

  • Perspective on current storage technology issue
  • Why this issue is unlikely to disappear
  • Open a discussion on how to standarize coverage for the issue

KEYNOTE SPEAKERS AND GENERAL SESSIONS

Cloud Architecture in the Data Center and the Impact on the Storage Ecosystem: A Journey Down the Software Defined Storage Rabbit Hole

Dan Maslowski, Global Engineering Head, Citi Storage/Engineered Systems - Citi Architecture and Technology Engineering (CATE), Citigroup

Abstract

Linux is usually at the edge of implementing new storage standards, and NVMe over Fabrics is no different in this regard. This presentation gives an overview of the Linux NVMe over Fabrics implementation on the host and target sides, highlighting how it influenced the design of the protocol by early prototyping feedback. It also tells how the lessons learned during developing the NVMe over Fabrics, and how they helped reshaping parts of the Linux kernel to support over Fabrics and other storage protocols better.


MarFS: A Scalable Near-POSIX File System over Cloud Objects for HPC Cool Storage

Grider, Gary, HPC Div Director, LANL

Abstract

Many computing sites need long-term retention of mostly cold data often “data lakes”. The main function of this storage tier is capacity but non trivial bandwidth/access requirements exist. For many years, tape was the best economic solution. Data sets have grown larger more quickly than tape bandwidth improvements and access demands have increased in the HPC environment. Disk can be more economically for this storage tier. The Cloud Community has moved towards erasure based object stores to gain scalability and durability using commodity hardware. The Object Interface works for new applications but legacy applications utilize POSIX for their interface. MarFS is a Near-POSIX File System using cloud storage for data and many POSIX file systems for metadata. Extreme HPC environments require that MarFS scale a POSIX namespace metadata to trillions of files and billions of files in a single directory while storing the data in efficient massively parallel ways in industry standard erasure protected cloud style object stores.

Learning Objectives

  • Storage tiering in future HPC and large scale computing environments
  • Economic drivers for implementing data lakes/tiered storage
  • HPC specific requirements for data lakes - multi way scaling
  • Overview of existing data lake solution space
  • How the MarFS solution works and for what types of situations

Networking

NVMe Over Fabrics Support in Linux

Christoph Hellwig

Abstract

Linux is usually at the edge of implementing new storage standards, and NVMe over Fabrics is no different in this regard. This presentation gives an overview of the Linux NVMe over Fabrics implementation on the host and target sides, highlighting how it influenced the design of the protocol by early prototyping feedback. It also tells how the lessons learned during developing the NVMe over Fabrics, and how they helped reshaping parts of the Linux kernel to support over Fabrics and other storage protocols better.

Object Storage

Hardware Based Compression in Ceph OSD with BTRFS

Weigang Li, Software Engineer, Intel
Anjaneya (Reddy) Chagam, Principle Engineer, Intel

Abstract

Ceph is a distributed object store and file system designed to provide excellent performance, reliability and scalability. BTRFS with the compelling set of features is recommended for non-production Ceph environments. In this talk we will introduce our experiment work on integrating the hardware acceleration in BTRFS to optimize the data compression workload in Ceph OSD, we analyze the nature of compression feature in BTRFS file system and the cost of the software compression library, and present the optimized solution to reduce the CPU cycles, disk IO with the hardware compression accelerator enabled in Ceph OSD.

Learning Objectives

  • BTRFS compression architecture
  • Hardware based compression acceleration integration with BTRFS
  • Ceph performance improvement with the hardware acceleration

Object Storage Analytics : Leveraging Cognitive Computing For Deriving Insights And Relationships

Sandeep Patil, Master Inventor, IBM
Pushkar Thorat, Software Engineer, IBM

Abstract

Object storage has become a de facto cloud storage for both private and public cloud deployment. Analytics over data stored on object store for deriving greater insights is an obvious exercise being looked by implementers. In object store where the data resides within objects, the user-defined metadata associated with the objects has the ability to provide quick relevant insights of the data. Hence leveraging user defined object metadata for analytics can help derive early insights. But having relevant user defined metadata with every object data is one of the biggest inhibitors for such analytics. On the other hand, Conginitive Computing has been an up trend where fiction meets reality. Various cognitive services are available which leverage extreme data analysis using machine learning techniques that help in data interpretation and beyond. In this presentation, we discuss on how cognitive services can help enrich object stores for analytics by self tagging objects which can not only be used for data analytics but also for deriving object relationships to helps short-list & categorize the data for analytics. The presentation includes a manifestation using IBM Spectrum Scale Object Store based on OpenStack SWIFT and popular cognitive service in marketplace like IBM Watson.

Learning Objectives

  • One of the objective is to understand how to analyze data on object store
  • Audience will learn about cognitive computation and how it relates to analytics
  • Audience will be able to understand how the emerging cognitive services can be applied to object store for better analysis of data hosted on object store
  • Audience will be able learn practically how they one can apply cognitive services for media based workloads hosted on object store to derive better insights

Fun with Linearity: How Encryption and Erasure Codes are Intimately Related

Jason Resch, Senior Software Architect, IBM

Abstract

Erasure codes are a common means to achieve availability within storage systems. Encryption, on the other hand, is used to achieve security for that same data. Despite the widespread use of both methods together, it remains little known that both of these functions are linear transformations of the data. This relation allows for them to be combined in useful ways. Ways that are seemingly unknown and unused in practice. This presentation presents novel techniques built on this observation, including: rebuilding lost erasure code fragments without exposing any information, decrypting fragments produced from encrypted source data, and verifying consistency and integrity of erasure coded fragments without exposing any information about the fragments or the data.

Learning Objectives

  • What are linear functions?
  • Examples of linear functions
  • Combining encryption and erasure codes
  • Exploiting the linearity of erasure codes to securely rebuild
  • Using the linearity of CRCs to securely verify erasure coded data

Storage Solutions for Private Cloud, Object Storage Implementation on Private Cloud, MAS/ACS

Ali Turkoglu, Principle Software Engineer Manager, Microsoft
Mallikarjun Chadalapaka, Principle PM, Microsoft

Abstract

In MAS (Microsoft Azure Stack), we implemented a scalable object storage on unified namespace leveraging Windows Server 2016 features in file system and in places creating new storage back ends for Azure block and page blobs, as well as table and queue services. This talk gives a details about capabilities of private cloud object storage and key design/architectural principles and problems being faced.

Learning Objectives

  • Private cloud object storage
  • Block and page blob design challenges
  • Microsoft Azure Stack

Dynamic Object Routing

Balaji Ganesan, Director Of Engineering, Cloudian

Abstract

Problem Description: In object store systems objects are identified using a unique identifier. Objects on disk are immutable. Any modification to an existing object, results in new object with different timestamp to be created on disk. A common method to distribute objects to storage nodes is using consistency hashing method. A storage node can have multiple disks to store the objects assigned to it and based on the application usage patterns some hash bucket grow faster than others resulting in some disks getting more used than others.

* Existing solution: - One way to solve this problem it to move this hash bucket to one of the less used disks, which moves data from disk to disk

* Our solution - A routing table is used to determine object's storage location. Object hash value as well as its insertion timestamp is used to determine the object's storage location. Each hash bucket is assigned initially to one of the available disks. When a hash bucket's storage disk utilization is greater than overall average disk utilization, another less used disk is assigned to that hash bucket with a new timestamp. All new objects to that hash bucket will be stored in new disk. Existing objects will be accessed from old disk using the routing table. This method will avoid moving data.

Open Source

Corporate/Open Source Community Relationships: The OpenZFS Example

Michael Dexter, Senior Analyst, iXsystems, Inc.

Abstract

Corporations and the global Open Source community have had a colorful relationship over the decades with each group struggling to understand the priorities, abilities and role of the other. Such relationships have ranged from hostile to prodigiously fruitful and have clearly resulted in the adoption of Open Source in virtually every aspect of computing. This talk will explore the qualities and precedences of strong Corporate/Open Source relationships and focus on the success of the OpenZFS enterprise file system project as a benchmark of contemporary success. I will explore:

  • Historical and contemporary corporate/open source relationship precedences
  • Corporation/Project non-profit foundation relationships
  • Pragmatic project collaboration and event participation strategies
  • Motivations for relationship building

 

Learning Objectives

  • How do I work with the Open Source community?
  • What organizations can I turn to for guidance and participation?
  • What tangible resources can the community provide?

Openstack

Introduction to OpenStack Cinder

Sean McGinnis, Sr. Principal Software Engineer, Dell
Walter Boring IV, Software Engineer, HPE

Abstract

Cinder is the block storage management service for OpenStack. Cinder allows provisioning iSCSI, fibre channel, and remote storage services to attach to your cloud instances. LVM, Ceph, and other external storage devices can be managed and consumed through the use of configurable backend storage drivers. Led by some of the core members of Cinder, this session will provide an introduction to the block storage services in OpenStack as well as give an overview of the Cinder project itself. Whether you are looking for more information on how to use block storage in OpenStack, are looking to get involved in an open source project, or are just curious about how storage fits into the cloud, this session will provide a starting point to get going.

Learning Objectives

  • Cloud storage management
  • OpenStack storage

Performance

IOPS: Changing Needs

Jim Handy, General Director, Objective Analysis
Thomas Coughlin, President, Coughlin Associates

Abstract

Four years have elapsed since our first IOPS survey – What has changed? Since 2012 we have been surveying SysAdmins and other IT professionals to ask how many IOPS they need and what latency they require. Things have changed over the past four years. Everyone understands that SSDs can give them thousands to hundreds of thousands of IOPS (I/Os Per Second), with flash arrays offering numbers in the million-IOPS range, while HDDs only support from tens to hundreds of IOPS. But many applications don’t need the extreme performance of high-end SSDs. In the survey users shared their IOPS needs with us both in 2012, and again in 2016. What has changed? What has remained the same? This presentation examines the need for high IOPS and profiles applications according to these needs.

Learning Objectives

  • Hear what your peers consider the necessary level of performance in both IOPS and latency for various common enterprise applications
  • Learn how the proper combination of HDDs and SSDS can satisfy IOPS and latency requirements for common enterprise activities
  • See some examples of how users have combined HDDs and flash memory to achieve cost effective solutions that meet their application requirements.

SPDK - Building Blocks for Scalable, High Performance Storage Applications

Benjamin Walker, Software Engineer, Intel Corporation

Abstract

Significant advances in throughput and latency for non-volatile media and networking require scalable and efficient software to capitalize on these advancements. This session will present an overview of the Storage Performance Development Kit, an open source software project dedicated to providing building blocks for scalable and efficient storage applications with breakthrough performance. There will be a focus on the motivations behind SPDK's userspace, polled-mode model, as well as details on the SPDK NVMe, CB-DMA, NVMe over Fabrics and iSCSI building blocks. http://spdk.io.

Learning Objectives

  • Why use userspace drivers
  • When polling is better than interrupts
  • Applying shared-nothing architecture to storage

Data Center Workload Analysis

Peter Murray, Principal System Engineer, Virtual Instruments

Abstract

For years, enterprises have desired a way to ensure that an application will operate properly at scale on a particular storage array and network infrastructure, and increasingly, ensure that multiple applications can safely run at capacity on an all-flash or hybrid architecture. Companies need an approach that enables users to reference detailed workload analysis profiles as they build and share workload models, and simulate performance across a variety of vendors, products and technologies.

Learning Objectives

  • How to determine what the traffic related to an enterprise application implementation looks like from the perspective of the network infrastructure between application servers and storage arrays
  • How the application I/O workload can be modeled to accurately represent the application in production using storage analytics or vendor performance statistics to generate a synthetic workload model
  • How a statistical application model can be used to validate both array and network performance at varying traffic levels and rates to rapidly test if/then scenarios
  • How multiple application models can be combined to determine appropriate throughput and response time levels on a given array
  • How using these models can enable engineers and architects to cost-effectively test and confidently deploy new networked storage infrastructure

Introducing the EDA Workload for the SPEC SFS® Benchmark

Nick Principe, Principal Software Engineer, EMC Corporation
Jignesh Bhadaliya, CTO for EDA, EMC Corporation

Abstract

The SPEC SFS subcommittee is currently finalizing an industry-standard workload that simulates the storage access patterns of large-scale EDA environments. This workload is based upon dozens of traces from production environments at dozens of companies, and it will be available as an addition to the SPEC SFS benchmark suite. Join us to learn more about the storage characteristics of real EDA environments, how we implemented the EDA workload in the SPEC SFS benchmark, and how this workload can help you evaluate the performance of storage solutions.

Persistent Memory

Enabling Remote Access to Persistent Memory on an IO Subsystem using NVM Express and RDMA

Stephen Bates, Sr. Technical Director, Microsemi

Abstract

NVM Express is predominately a block based protocol where data is transferred to/from host memory using DMA engines inside the PCIe SSD. However since NVMe 1.2 there exists a memory access method called a Controller Memory Buffer which can be thought of as a PCIe BAR managed by the NVMe driver. Also, the NVMe over Fabrics standard was released this year that extended NVMe over RDMA transports. In this paper we look at the performance of the CMB access methodology over RDMA networks. We discuss the implications of adding persistence semantics to both RDMA and these NVMe CMBs to enable a new type of NVDIMM (which we refer to as an NVRAM). These NVRAMs can reside on the IO sub-system and hence are decoupled from the CPU and memory sub-system which has certain advantages and disadvantages over NVDIMM which we outline in the paper. We conclude with a discussion on how NVMe over Fabrics might evolve to support this access methodology and how the RDMA standard is also developing to align with this work.

Learning Objectives

  • NVM Express overview
  • NVMe over Fabrics overview
  • Using IO memory as an alternative to NVDIMM
  • Controller Memory Buffers in NVMe
  • Performance results for new access method

RDMA Extensions for Accelerating Remote PMEM Access - HW and SW Considerations, Architecture, and Programming Models

Chet Douglas, Principal SW Engineer, Intel Corporation
Raj Ramanujan, Sr. Principal Engineer, Intel Corporation

Abstract

The SNIA Nonvolatile Memory Programing Model defines recommendations on how NVM behavior can be exposed to application software. There are Linux NVM library implementations that provide a rich set of interfaces which can be used by applications to improve their performance drastically on systems with NVM.

We analyzed these available interfaces and how it can be leveraged in typical applications. Our work demonstrates how a sample OpenSource Linux application can make use of these interfaces for improving performance. We also give examples of analyzing application code and finding out opportunity to use NVMp style interfaces. The sample application we would discuss is Linux Sqllite, and found 9 such opportunities to use Linux NVMP compatible interfaces. We also show how these storage optimizations can improve overall I/O and performance of such applications.

Learning Objectives

  • Introduce HW Architecture concepts of Intel platforms that will affect RDMA usages with PM
  • Introduce proposed high-level HW modifications that can be utilized to provide native HW support for pmem, reduce RDMA latency, and improve RDMA with pmem bandwidth
  • Focused discussion on proposed Linux libfabric and libibverb interface extensions and modification to support the proposed HW extensions Discuss open architecture issues and limitations with the proposed HW and SW extensions
  • Discuss Intel plans for standardization and industry review

Experience and Lessons from Accelerating Linux Application Performance Using NVMp

Vinod Eswaraprasad, Chief Architect, Wipro Technologies

Abstract

The SNIA Nonvolatile Memory Programing Model defines recommendations on how NVM behavior can be exposed to application software. There are Linux NVM library implementations that provide a rich set of interfaces which can be used by applications to improve their performance drastically on systems with NVM.

We analyzed these available interfaces and how it can be leveraged in typical applications. Our work demonstrates how a sample OpenSource Linux application can make use of these interfaces for improving performance. We also give examples of analyzing application code and finding out opportunity to use NVMp style interfaces. The sample application we would discuss is Linux Sqllite, and found 9 such opportunities to use Linux NVMP compatible interfaces. We also show how these storage optimizations can improve overall I/O and performance of such applications.

Learning Objectives

  • NVMp - programming model and use cases
  • Linux NVMP implementations
  • Analyzing application for usage of NVM
  • Measure performance improvements in application - with specific workload focusing on I/O

Breaking Barriers: Making Adoption of Persistent Memory Easier

Sarah Jelinek, SW Architect, Intel Corp

Abstract

One of the major barriers to adoption of persistent memory is preparing applications to make use of it's direct access capabilities. This presentation will discuss a new user space file system for persistent memory and how it breaks these barriers. The presentation will introduce the key elements to consider for a user space persistent memory file system and discuss the internals of this new file system. The discussion will conclude with a presentation of current status and performance of this new persistent memory file system.

Learning Objectives

  • Discussion of current barriers to persistent memory adoption.
  • Introduce how this new file system breaks down the barriers to adoption of persistent memory.
  • Introduce the SW internals of the this file system.
  • Present performance statistics and discussion of why this file system out-performs conventional, kernel based file systems.

Building on The NVM Programming Model – A Windows Implementation

Paul Luse, Principal Engineer, Intel
Chandra Kumar Konamki, Sr Software Engineer, Microsoft

Abstract

In July 2012 the SNIA NVM Programming Model TWG was formed with just 10 participating companies who set out to create specifications to provide guidance for operating system, device driver, and application developers on a consistent programming model for next generation non-volatile memory technologies. To date, membership in the TWG has grown to over 50 companies and the group has published multiple revisions of The NVM Programming Model. Intel and Microsoft have been long time key contributors in the TWG and we are now seeing both Linux and Windows adopt this model in their latest storage stacks. Building the complete ecosystem requires more than just core OS enablement though; Intel has put considerable time and effort into a Linux based library, NVML, that adds value in multiple dimensions for applications wanting to take advantage of persistent byte addressable memory from user space. Now, along with Intel and HPE, Microsoft is moving forward with its efforts to further promote this library by providing a Windows implementation with a matching API. In this session you will learn the fundamentals of the programming model, the basics of the NVML library and get the latest information on the Microsoft implementation of this library. We will cover both available features/functions and timelines as well as provide some insight into how the open source project went from idea to reality with great contributions from multiple companies.

Learning Objectives

  • NVM Programming Model Basics
  • The NVM Libraries (NVML)
  • The Windows Porting Effort

Low Latency Remote Storage - a Full-stack View

Tom Talpey, Architect, Microsoft

Abstract

A new class of ultra low latency remote storage is emerging - nonvolatile memory technology can be accessed remotely via high performance storage protocols such as SMB3, over high performance interconnects such as RDMA. A new ecosystem is emerging to "light up" this access end-to-end. This presentation will explore one path to achieve it, with performance data on current approaches, analysis of the overheads, and finally the expectation with simple extensions to well-established protocols.

Learning Objectives

  • Understand the potential for low latency remote storage
  • Survey the protocols and interfaces in use today
  • See current performance data, and future performance expectations
  • See a view of the future of the end-to-end storage revolution

The SNIA NVM Programming Model

Doug Voigt, Distinguished Technologist, Hewlett Packard Enterprise

Abstract

The SNIA NVM Programming model enables applications to consume emerging persistent memory technologies through step-wise evolution to greater and greater value. Starting with an overview of the latest revision of the NVM programming model specification this session summarizes the recent work of the NVM programming TWG in areas of high availability and atomicity. We take an application view of ongoing technical innovation in a persistent memory ecosystem.

Learning Objectives

  • Learn what the SNIA NVM programming TWG has been working on.
  • Learn how applications can move incrementally towards greater and greater benefit from persistent memory.
  • Learn about the resources available to help developers plan and implement persistent memory aware software.

Challenges with Persistent Memory in Distributed Storage Systems

Dan Lambright, Principal Software Engineer, Red Hat

Abstract

Persistent memory will significantly improve storage performance. But these benefits can be harder to realize in distributed storage systems such as Ceph or Gluster. In such architectures, several factors mitigate gains from faster storage. They include costly network overhead inherent for many operations, and deep software layers supporting necessary services. It is also likely that the high costs of persistent memory will limit deployments in the near term. This talk shall explore steps to tackle those problems. Strategies include tiering subsets of the data or metadata in persistent memory, incorporating high speed networks such as infiniband, and the use of tightly optimized “thin” file systems. The talk will include discussion and results from a range of experiments in software and hardware.

Learning Objectives

  • Understand bottlenecks in scale-out storage when persistent memory is used
  • Learn the benifits and limits of faster networks in such configurations
  • Understand the current stacks and where improvements can be made

In-memory Persistence: Opportunities and Challenges

Dhruva Chakrabarti, Research Engineer, Hewlett Packard Labs

Abstract

The advent of persistent memory enables a new class of applications where objects can be persisted in-memory as well as reused across executions and applications. This storage paradigm gets rid of the impedance mismatch between in-memory and stored data formats that is inherent using today’s block storage. A single (object) format of data should result in both performance and programmability benefits.

However, it is far from clear how to program persistent memory in a failure-tolerant manner. Opening up persistence to arbitrary data structures implies that any failure-tolerance technique should be largely transparent to the programmer while incurring low overheads. In this talk, I will review prior work on persistent programming paradigms and describe some recent work that provides consistency support with zero to minimum code changes. The audience can expect to learn about a specific model and its APIs. Other challenges and possible future directions will be discussed.

Learning Objectives

  • Overview of persistent memory and examples of upcoming platforms surrounding it
  • The existing literature on persistent memory programming
  • In-memory persistence and associated consistency pitfalls
  • An example failure-resilient persistent memory programming platform

There and Back Again, a Practical Presentation on Using Storage Class Memory With Applications

Mat Young, VP of Marketing, Netlist
Chad Dorton, App Architect, Netlist

Abstract

Storage class memory (SCM) fills the gap between DRAM and NVMe SSD. SCM has the ability to deliver performance at-or-near the performance of DRAM with the persistence of flash. In order to realize these gains, however, software stacks must be updated to take advantage of SCM in a more native fashion. In this session we will demonstrate how recent advancements in the Linux Kernel like DAX and libpmem allow firms to take of the speed of SCM without any modification of the application. Go beyond bandwidth and IOPS to the real world of application performance acceleration with Storage Class Memory.

Learning Objectives

  • Myth busting on NVM
  • Show the real world beyond IOPS
  • Explanation and Demo of DAX
  • Explanation and demo of using memoy mapped files
  • Discussion and demo of appliction modifed to use NVML

Persistent Memory Quick Start Programming Tutorial

Jeff Sebby, Solution Architect, Intel Corporation

Abstract

A tutorial on using the open source persistent memory (PMEM) programming library offered at pmem.io; including examples providing power-fail safety, while storing data on PMEM. Programming examples will zero in on pmem.io’s transactional object store (i.e. “libpmemobj”) library, which is layered on the SNIA NVM Programming Model. The examples covered will demonstrate proper integration techniques, macros, C API, key theory concepts, terminology, and present in depth overview of what the library offers for PMEM programming initiatives.

Learning Objectives

  • Overview of persistent memory programming
  • Close examination of the transactional object store library (libpmemobj)
  • Code integration walkthrough

SMB

SMB3 and Linux - A Seamless File Sharing Protocol

Jeremy Allison, Engineer, Samba Team/Google

Abstract

SMB3 is the default Windows and MacOS X file sharing protocol, but what about making it the default on Linux ? After developing the UNIX extensions to the SMB1 protocol, the Samba developers are planning to add UNIX extensions to SMB3 also. Co-creator of Samba Jeremy Allison will discuss the technical challenges faced in making SMB3 into a seamless file sharing protocol between Linux clients and Samba servers, and how Samba plans to address them. Come learn how Samba plans to make NFS obsolete (again :-) !

Learning Objectives

  • SMB3
  • Linux
  • Windows interoperability

Security

When Will Self-Encrypting Drives Predominate?

Tom Coughlin, President, Coughlin Associates, Inc.
Walt Hubis, President, Hubis Consulting

Abstract

Self-Encrypting Drives (SEDs) have found applications in enterprise storage where crypto-erase allows rapid sanitization of retired drives but their use in client storage devices is still minimal. Learn about the history and uses for SEDs and what needs to happen to bring them into broader use based upon results from a recent poll of client end users.

Learning Objectives

  • What are SEDs used for?
  • What does a recent survey show are factors in client SED success?
  • What is needed to make SEDs widespread?

SNIA Tutorial:
Implementing Stored-Data Encryption

Michael Willett, VP Marketing, Drive Trust Alliance

Abstract

Data security is top of mind for most businesses trying to respond to the constant barrage of news highlighting data theft, security breaches, and the resulting punitive costs. Combined with litigation risks, compliance issues and pending legislation, companies face a myriad of technologies and products that all claim to protect data-at-rest on storage devices. What is the right approach to encrypting stored data?

The Trusted Computing Group, with the active participation of the drive industry, has standardized on the technology for self-encrypting drives (SED): the encryption is implemented directly in the drive hardware and electronics. Mature SED products are now available from all the major drive companies, both HDD (rotating media) and SSD (solid state) and both laptops and data center. SEDs provide a low-cost, transparent, performance-optimized solution for stored-data encryption. SEDs do not protect data in transit, upstream of the storage system.

For overall data protection, a layered encryption approach is advised. Sensitive data (eg, as identified by specific regulations: HIPAA, PCI DSS) may require encryption outside and upstream from storage, such as in selected applications or associated with database manipulations.

This tutorial will examine a ‘pyramid’ approach to encryption: selected, sensitive data encrypted at the higher logical levels, with full data encryption for all stored data provided by SEDs.

Learning Objectives

  • The mechanics of SEDs, as well as application and database-level encryption
  • The pros and cons of each encryption subsystem
  • The overall design of a layered encryption approach

SMR

ZDM: Using an STL for Zoned Media on Linux

Shaun Tancheff, Software Engineer, AeonAzure LLC

Abstract

As zoned block devices supporting the T10 ZBC and T13 ZAC specifications are becoming available there few strategies for putting these low TCO ($/TB and watts/TB) drives into existing storage clusters with minimal changes to the existing software stack.

ZDM is a Linux device mapper target for zoned media that provides a Shingled Translation Layer (STL) to support a normal block interface at near normal performance for certain use cases.

ZDM-Device-Mapper is an open source project available on github with a goal of being upstreamed to the kernel.

Learning Objectives

  • ZDM performance compared to existing disc drive options
  • How to determine if your workload is a likely candidate for using ZDM.
  • How to use and tune ZDM effectively for your workload.
  • Using ZDM in cloud storage environments.
  • Using ZDM in low resource embedded NAS / mdraid.

ZBC/ZAC Support in Linux

Damien Le Moal, Sr. Manager, Western Digital

Abstract

With the support for the ZBC and ZAC shingled magnetic recording (SMR) command standards maturing in Linux kernel, application developers can now more easily implement support for these emerging high-capacity disks. This presentation will go through the basics of SMR support in Linux kernel and provide an overview of different application implementation approaches, including the use of available SMR abstraction layers (file system and device mappers).

Learning Objectives

  • Linux kernel SMR disk management
  • Applications SMR constraints
  • SMR abstraction layers

The Shingled Magnetic Recording (SMR) Revolution – Data Management Techniques Examined

Tom Coughlin, President, Coughlin Associates

Abstract

The unyielding growth of digital data continues to drive demand for higher capacity, lower-cost storage. With the advent of Shingled Magnetic Recording (SMR), which overlaps HDD tracks to provide a 25 percent capacity increase versus conventional magnetic recording technology, storage vendors are able to offer extraordinary drive capacities within existing physical footprints. That said, IT decision makers and storage system architects need to be cognizant of the different data management techniques that come with SMR technology, namely Drive Managed, Host Managed and Host Aware. This panel session will offer an enterprise HDD market overview from prominent storage analyst Tom Coughlin as well as presentations on SMR data management methods from leading SMR HDD manufacturers (Seagate, Toshiba and Western Digital).

Learning Objectives

  • Enterprise HDD market update
  • Brief introduction of Shingled Magnetic Recording (SMR)
  • Deep dive into SMR data management techniques

Software Defined Storage

Storage Spaces Direct - A Software Defined Storage Architecture

Vinod Shankar, Principal Development Lead, Microsoft Corporation

Abstract

Storage Spaces Direct is Microsoft’s Software Defined Storage architecture for the private cloud. This talk will cover the technical architecture of S2D and its evolution from our SDC 2015 presentation to the final form for Windows Server 2016. We will discuss implementation of higher order fault domains, technical details and rationale for multi-resilient virtual disks that blend mirror and parity encoding, stretch clusters with synchronous replication and more. Software defined storage solutions can (and should still!) be based on industry-standard hardware!

Learning Objectives

  • Software defined storage on windows
  • Hybrid storage

Solid State Storage

Standards for Improving SSD Performance and Endurance

Bill Martin, Principal Engineer Storage Standards, Samsung

Abstract

Standardization efforts have continued for features to improve SSD performance and endurance. NVMe, SCSI, and SATA have completed standardization of streams and background operation control. Standardization is beginning on how SSDs may implement Key Value Storage (KVS) and In Storage Compute (ISC). This effort is progressing in the SNIA Object Drive Technical Work Group (TWG), and may involve future work in the protocol standards (NVMe, SCSI, and SATA). Currently the Object Drive TWG is defining IP based management for object drives in utilizing the DMTF RedFish objects. Future Object Drive TWG work will include APIs for KVS and ISC. This presentation will discuss the standardization of streams, background operation control, KVS, and ISC and how each of these features work.

Learning Objectives

  • What is streams
  • What is background operation control
  • Progress of standardization in NVMe, SCSI, SATA, and SNIA

Breaking Through Performance and Scale Out Barriers - A Storage Solution for Today's Hot Scale Out Applications

Bob Hansen, V.P. Systems Architecture, Apeiron Data Systems

Abstract

This presentation will review how classic storage architectures create barriers to throughput and scaleability for several of today's hot applications. Case studies and benchmarks will be used to illustrate how a very high performance, NVMe networked storage solution allows these applications to break through these barriers.

Learning Objectives

  • Understand how storage can limit scale out application performance
  • Understand how new scale out apps apply different tiers of storage
  • Understand barriers to application scale out
  • Understand barriers to application throughput
  • Understand how networked NVMe storage can dramatically improve app performance and scalability

Storage Architecture

Heterogeneous Architectures for Implementation of High-capacity Hyper-converged Storage Devices

Michaela Blott, Principal Engineer, Xilinx/Research
Endric Schubert, PhD CTO & Founder, MLE

Abstract

Latest trends in software defined storage indicate emergence of hyper-convergence where compute networking and storage are combined within one device. In this talk, we introduce a novel architecture to implement such a node in one device. The unique features include a combination of ARM-processors for control plane functionality and dataflow architectures in FPGAs to handle data processing. We leverage a novel hybrid memory system mixing NVMe drives and DRAM to deliver a multi-terabyte object store with 10Gbps access bandwidth. Finally, network connectivity is accelerated by leveraging a full TCP/IP endpoint dataflow implementation within the FPGA’s programmable fabric. A first proof-of-concept deploys Xilinx Ultrascale+ MPSoC to demonstrate feasibility of a single-chip solution that can produce unprecedented levels of performance (13M requests per second) and storage capacity (2TB) at minimal power consumption (<30W). Furthermore, the deployed dataflow architecture supports additional software-defined features such as video compression and object recognition without performance penalty while resources fit in the device.


The Magnetic Hard Disk Drive Today’s Technical Status and Its Future

Edward Grochowski, Consultant, Memory/Storage
Peter Goglia, President, VeriTekk Solutions

Abstract

The ubiquitous magnetic hard disk drive continues to occupy a principal role in all storage applications, shipping more bytes than all other competing product technologies. The emergence of cloud computing has firmly established the future HDD products well into the next decade. This presentation will discuss today’s HDD products, their technical characteristics, as; performance, reliability and endurance, capacity as well as cost per gigabyte. The technologies which enable these properties as; form factor, interface, shingle write, helium ambient, two dimensional magnetic recording (TDMR) and yet to be implemented heat assisted magnetic recording (HAMR), will be detailed. A projection of disk drives of the future will be made and their competiveness to flash as well as emerging non volatile memories (NVM) will be discussed.


What One Billion Hours of Spinning Hard Drives Can Tell Us?

Gleb Budman, CEO, Backblaze

Abstract

Over the past 3 years we’ve been collecting daily SMART stats from the 60,000+ hard drives in our data center. These drives have over one billion hours of operation on them. We have data from over 20 drive models from all major hard drive manufacturers and we’d like to share what we’ve learned. We’ll start with annual failure rates of the different drive models. Then we’ll look at the failure curve over time, does it follow the “bathtub curve” as we expect. We’ll finish by looking a couple of SMART stats to see if they can reliably predict drive failure.

Learning Objectives

  • What is the annual failure rate of commonly used hard drives?
  • Do hard drives follow a predictable pattern of failure over time?
  • How reliable are drive SMART stats in predicting drive failure?

Reducing Replication Bandwidth for Distributed Document-oriented Databases

Sudipta Sengupta, Principal Research Scientist, Microsoft Research

Abstract

With the rise of large-scale, Web-based applications, users are increasingly adopting a new class of document-oriented database management systems (DBMSs) that allow for rapid prototyping while also achieving scalable performance. As in other distributed storage systems, replication is important for document DBMSs in order to guarantee availability. The network bandwidth required to keep replicas synchronized is expensive and is often a performance bottleneck. As such, there is a strong need to reduce the replication bandwidth, especially for geo-replication scenarios where wide-area network (WAN) bandwidth is limited. This talk presents a deduplication system called sDedup that reduces the amount of data transferred over the network for replicated document DBMSs. sDedup uses similarity-based deduplication to remove redundancy in replication data by delta encoding against similar documents selected from the entire database. It exploits key characteristics of document-oriented workloads, including small item sizes, temporal locality, and the incremental nature of document edits. Our experimental evaluation of sDedup with real-world datasets shows that it is able to significantly outperform traditional chunk-based deduplication techniques in reducing data sent over the network while incurring negligible performance overhead.

Learning Objectives

  • Replication in distributed databases
  • Techniques for network bandwidth reduction
  • Similarity detection in sDedup
  • sDedup system design

Open Storage Platform Delivers Hyper-scale Benefits for the Enterprise

Eric Slack, Sr. Analyst, Evaluator Group

Abstract

The Open Storage Platform (OSP) is an architecture for building storage and compute infrastructures using independent software and hardware components that leverages some popular concepts in IT – scale-out storage, software-defined storage and commodity hardware. This approach originated in the large, hyper-scale companies as a way to provide the enormous amounts of storage capacity they needed, cost effectively.

These companies popularized the use of low-cost server chassis filled with consumer-grade storage media. They also used complex software, often developed in-house, to provide resiliency at a cluster level and federate the large numbers of physical servers that comprised these infrastructures.

The Open Storage Platform provides an architecture for assembling these scale-out storage systems from commercially available products. Where the hyper-scalers’ experience was a “do-it-yourself” kind of process, OSP could be described as more “roll your own”.

Learning Objectives

  • Understand the OSP model and its origin
  • Understand where OSP is being used Understand the benefits OSP can provide
  • Understand how to implement OSP

SNIA Tutorial:
Your Cache is Overdue a Revolution: MRCs for Cache Performance and Isolation

Irfan Ahmad, CTO, CloudPhysics

Abstract

It is well-known that cache performance is non-linear in cache size and the benefit of caches varies widely by workload. Irrespective of whether the cache is in a storage system, database or application tier, no two real workload mixes have the same cache behavior! Existing techniques for profiling workloads don’t measure data reuse, nor do they predict changes in performance as cache allocations are varied.

Recently, a new, revolutionary set of techniques have been discovered for online cache optimization. Based on work published at top academic venues (FAST '15 and OSDI '14), we will discuss how to 1) perform online selection of cache parameters including cache block size and read-ahead strategies to tune the cache to actual customer workloads, 2) dynamic cache partitioning to improve cache hit ratios without adding hardware and finally, 3) cache sizing and troubleshooting field performance problems in a data-driven manner. With average performance improvements of 40% across large number of real, multi-tenant workloads, the new analytical techniques are worth learning more about.

Learning Objectives

  • Storage cache performance is non-linear; Benefit of caches varies widely by workload mix
  • Working set size estimates don't work for caching Miss ratio curves for online cache analysis and optimization
  • How to dramatically improve your cache using online MRC, partitioning, parameter tuning
  • How to implement QoS, performance SLAs/SLOs in caching and tiering systems using MRCs

An Enhanced I/O Model for Modern Storage Devices

Martin Petersen, Architect, Oracle

Abstract

While originally designed for disk drives, the read/write I/O model has provided a common storage abstraction for decades, regardless of the type of storage medium.

Devices are becoming increasingly complex, however, and the constraints of the old model have compelled the standards bodies to develop specialized interfaces such as the Object Storage Device and the Zoned Block Commands protocols to effectively manage the storage.

While these protocols have their place for certain workloads, there are thousands of filesystems and applications that depend heavily on the old model. It is therefore compelling to explore how the read/write mechanism can be augmented using hints and stream identifiers to communicate additional information that enables the storage to make better decisions.

The proposed model is applicable to all types of storage devices and alleviates some of the common deficiencies with NAND flash and Shingled Magnetic Recording which both require careful staging of writes to media.

Learning Objectives

  • Present the proposed enhancements to a wider audience
  • Solicit feedback and nurture discussion
  • Demonstrate how the proposed enhancements reduce write amplification on flash and assist with data placement on SMR

SNIA Tutorial:
Snapshotting Eventually Consistent Scale-out Storage: The Pitfalls and Solutions

Alex Aizman, CTO & Founder, Nexenta Systems

Abstract

The most difficult to support is the requirement of consistency that often implies not only storage system’s own internal consistency (which is mandatory) but the application level consistency as well. Distributed clocks are never perfectly synchronized: temporal inversions are inevitable. While most I/O sequences are order indifferent, we cannot allow inconsistent snapshots that reference user data contradicting the user’s perception of ordering. Further, photo snapshots do not require the subjects to freeze: distributed snapshot operation must not require cluster-wide freezes. It must execute concurrently with updates and eventually (but not immediately) result in a persistent snapshot accessible for reading and cloning. This presentation will survey distributed snapshotting, explain and illustrate ACID properties of the operation and their concrete interpretations and implementations. We will describe MapReduce algorithms to snapshot a versioned eventually consistent object cluster. Lastly the benefits of atomic cluster-wide snapshots for distributed storage clusters will be reviewed. True cluster-wide snapshots enable capabilities and storage services that had been feared lost when storage systems scaled-out beyond transactional consistency of medium-size clusters.

Learning Objectives

  • Distributed copy-on-write snapshots - a lost art?
  • Usage of client-defined timestamps to support causal consistency
  • MapReduce programming model vs. cluster-wide snapshotting – a perfect fit

Storage Management

Overview of Swordfish: Scalable Storage Management

Richelle Ahlvers, Principal Storage Management Architect, Broadcom Limited

Abstract

The SNIA’s Scalable Storage Management Technical Work Group (SSM TWG) is working to create and publish an open industry standard specification for storage management that defines a customer centric interface for the purpose of managing storage and related data services. This specification builds on the DMTF’s Redfish specification’s using RESTful methods and JSON formatting. This session will present an overview of the specification being developed by SSM including the scope targeted in the initial (V1) release in 2016 vs later (2017). This session will also provide the positioning of the specification developed by the SSM TWG vs SMI-S as well as the base Redfish specification.


Swordfish Deep-dive: Scalable Storage Management

Richelle Ahlvers, Principal Storage Management Architect, Broadcom Limited

Abstract

Building on the concepts presented in the Introduction to Swordfish (and Redfish) sessions, this session will go into more detail on the new Swordfish API specification.

Learning Objectives

  • Introduction to the specifics of the Swordfish API
  • Working with the Swordfish Schema

Introduction and Overview of Redfish

Richelle Ahlvers, , Principal Storage Management Architect, Broadcom Limited
Jeff Autor, HP Enterprise

Abstract

Designed to meet the expectations of end users for simple, modern and secure management of scalable platform hardware, the DMTF’s Redfish is an open industry standard specification and schema that specifies a RESTful interface and utilizes JSON and OData to help customers integrate solutions within their existing tool chains.

This session provides an overview of the Redfish specification, including the base storage models and infrastructure that are used by the SNIA Swordfish extensions (see separate sessions for details).

We will cover details of the Redfish approach, as well as information about the new PCIe and memory models added to support storage use cases.

Learning Objectives

  • Introduction to Redfish concepts
  • Application of REST APIs to standards management

Storage Networking

SNIA Tutorial:
SAS: Today’s Fast and Flexible Storage Fabric

Rick Kutcipal, President, SCSI Trade Association, Product Planning, Data Center Storage SCSI Trade Association, Broadcom

Abstract

For storage professionals seeking fast, flexible and reliable data access, Serial Attached SCSI (SAS) is the proven platform for innovation. With a robust roadmap, SAS provides superior enterprise-class system performance, connectivity and scalability. This presentation will discuss why SCSI continues to be the backbone of enterprise storage deployments and how it continues to rapidly evolve by adding new features, capabilities, and performance enhancements. It will include an up-to-the-minute recap of the latest additions to the SAS standard and roadmaps, the status of 12Gb/s SAS deployment, advanced connectivity solutions, MultiLink SAS™, and 24Gb/s SAS development. Presenters will also provide updates on new SCSI features such as Storage Intelligence and Zoned Block Commands (ZBC) for shingled magnetic recording.

Learning Objectives

  • Understand the basics of SAS architecture and deployment, including its compatibility with SATA, that makes SAS the best device level interface for storage devices.
  • Hear the latest updates on the market adoption of 12Gb/s SAS and why it is significant. See high performance use case examples in a real-world environment such as distributed databases.
  • See examples of how SAS is a potent connectivity solution especially when coupled with SAS switching solutions. These innovative SAS configurations become a vehicle for low cost storage expansion

Testing

Uncovering Distributed Storage System Bugs in Testing (not in Production!)

Shaz Qadeer, Principal Researcher, Microsoft
Cheng Huang, Senior Researcher, Microsoft

Abstract

Testing distributed systems is challenging due to multiple sources of nondeterminism. Conventional testing techniques, such as unit, integration and stress testing, are ineffective in preventing serious but subtle bugs from reaching production. Formal techniques, such as TLA+, can only verify high-level specifications of systems at the level of logic-based models, and fall short of checking the actual executable code. In this talk, we present a new methodology for testing distributed systems. Our approach applies advanced systematic testing techniques to thoroughly check that the executable code adheres to its high-level specifications, which significantly improves coverage of important system behaviors.

Our methodology has been applied to three distributed storage systems in the Microsoft Azure cloud computing platform. In the process, numerous bugs were identified, reproduced, confirmed and fixed. These bugs required a subtle combination of concurrency and failures, making them extremely difficult to find with conventional testing techniques. An important advantage of our approach is that a bug is uncovered in a small setting and witnessed by a full system trace, which dramatically increases the productivity of debugging.

Learning Objectives

  • Specifying distributed storage systems
  • Testing distributed storage systems
  • Experience with advanced testing techniques on distributed storage systems

Virtualization

Hyper-V Windows Server 2016 Storage QoS and Protocol Updates

Matt Kurjanowicz, Senior Software Engineer Lead, Microsoft

Abstract

A discussion of the Hyper-V Storage Quality of Service protocol (MS-SQOS) and updates to networked storage requirements for Windows Server 2016 virtual machine hosts. This session is targeted at engineers and product managers of providers of networked storage to Hyper-V hosts and anyone else interested in how Hyper-V hosts utilize storage. Those attending this session should be able to gain familiarity with customer scenarios around storage quality of service, should be able to describe the purpose and scope of the MS-SQOS protocol, and should understand how Windows Server 2016 made changes to how it uses networked storage.

Learning Objectives

  • Describe the purpose and scope of the MS-SQOS protocol
  • Become aware of customer scenarios around storage quality of service
  • Enumerate updates in how Hyper-V hosts use networked storage in Windows Server 2016