10 Gigabit Ethernet in the Data Center
by Jason Blosil, IPSF Member, NetApp and David Dale, Chair of IPSF

Over the past 5 years, we have seen a huge growth in the use of Ethernet as a storage fabric. Network File System (NFS) deployment has expanded from its Unix file sharing and technical applications roots into large-scale enterprise applications environments, particularly those benefiting from a file system interface for application administration, and data protection offloading from the host environment. Common Internet File Systems (CIFS) and Server Message Block (SMB) deployments have expanded as windows file-sharing has become increasingly important in the typical distributed IT environment. And finally, iSCSI has seen broad adoption in SAN solutions supporting business applications - particularly in Windows, Linux and VMware environments.

Gigabit Ethernet has proven to be very effective in addressing the majority of I/O requirements of these applications. And now the next generation of Ethernet, with 10 Gigabit per second speeds and feature enhancements around data center bridging, is accelerating the move to consolidate IT resources and reduce cost.

Ratified in June of 2002 as the IEEE 802.3ae standard, 10 Gigabit Ethernet (10 GbE) was initially targeted as an inter-switch link in high speed networks and Metropolitan Area Networks (MANs). However, as per port prices continued to drop, and server host architectures were enhanced to accommodate high-bandwidth I/O, 10 GbE is beginning to experience broad adoption as a storage fabric. Today, in fact, companies of all sizes are benefiting from the increased bandwidth and simplified management that 10 GbE has to offer.

Ethernet as a Storage Fabric
Ethernet as a storage fabric offers a number of benefits which simplify deployment and management, and improve performance and efficiency. These benefits include:

• Block and File Access - Ethernet supports block and file protocols over the same wire, including NFS, CIFS, and iSCSI
• Low Acquisition Cost - Ethernet is ubiquitous in the data center for LAN traffic duty - volumes are high and ports are embedded in processor chipsets
• Reliable, Interoperable - Ethernet over the past 25 years has demonstrated mission critical robustness and plug-and-play interoperability
• Ease of Management - Leverages a broadly available Ethernet skill set for configuration and maintenance; no need to hire or train dedicated storage staff
• Long Cable Distances - IP routing facilitates almost limitless distances for storage traffic
• Quality of Service - QOS is an inherent benefit of the TCP/IP layer which guarantees the delivery of data packets within a certain time frame and with the ability to ensure that the right order is maintained for packet delivery
• Secure - VLANs can be configured to logically isolate one LAN from another across the same wire; iSCSI supports Challenge-Handshake Authentication Protocol (CHAP) authentication and Internet Protocol Security (IPSec) encryption
• Virtual Addressing - Ethernet supports virtual addressing and dynamic routing through TCP/IP. Unlike some other network protocols, the logical address of an Ethernet device is not statically tied to a physical device. Virtual addressing is particularly interesting for use in virtual server environments where multiple hypervisors may be moved from one physical server to another in order to load balance or facilitate the servicing of hardware.

For SAN applications over Ethernet, iSCSI offers a number of advantages. iSCSI traffic runs on standard Ethernet hardware, including switches and embedded ports on servers, which are less expensive than special purpose storage network components. And iSCSI traffic can share existing Ethernet infrastructure, facilitating network consolidation and simplifying installation and manageability. Additionally, iSCSI offers increased flexibility and reduced complexity at the host with the option of using software initiators available with the host operating system, software initiators and some kind of TCP/IP offload solution, or iSCSI HBAs and drivers. At 1Gb/s Ethernet, application server workloads for typical enterprise applications are adequately handled by a software initiator solution (this is true in more than 85% of deployments today). For more demanding workloads, some kind of TCP/IP offload solution is often deployed. For 10Gb/s Ethernet, the expectation is that offload solutions will be more common, and some operating systems already offer the capability to process network traffic on a separate CPU core or on a special-purpose adapter, while preserving the simplicity of using the host OS-resident software initiator.

10 GbE for Storage and Server Consolidation
10 GbE offers an ideal fabric for both storage and server consolidation. As iSCSI based SANs have grown in size, there has been an increasing need for consolidation within the fabric. 10 GbE connections for enterprise storage arrays and 10/1Gb/s Ethernet switch solutions have been available for some time (1-2 years depending on vendor) - enabling a large 10GbE-connected array to support a large number of 1GbE-connected hosts. In addition, affordable 10GbE ports on standard servers have now become available, driving the server consolidation side of the equation.

Server consolidation is a rapidly-growing trend - particularly in environments that have seen the proliferation of low cost servers each running just a few applications. CPU utilization in these environments tends to be very low; manageability is low; and power consumption is very high. In practice, server consolidation (reducing the physical number of servers) is often accompanied by the move to a virtual server environment, which can significantly improve application provisioning and ease of management. However, this can greatly increase the I/O requirements to the physical server. 10 GbE is an excellent solution, providing plenty of I/O headroom, and supporting the port virtualization required to optimally support the deployment flexibility demanded by the virtual server environment.

Consolidated Network Fabric Benefits
10 Gigabit per second Ethernet deployments in the backbone of many IT data communications environments are already well under way. This deployment is not just in the network closet, but also in the data center, and over time, every IT organization will have some experience with this technology.

An increasing number of companies are deciding on Ethernet as their primary storage fabric for particular data centers, finding that NFS, CIFS and iSCSI meet all the application I/O requirements for that data center. 10 Gb/s Ethernet is often deployed in these environments too. So, it seems obvious that 10 GbE as a 'unified" network for data communications and for storage traffic is rapidly becoming a reality in some environments.

This trend is likely to accelerate with the availability of Fibre Channel over Ethernet (FCoE) in the coming year. FCoE enables existing Fibre Channel endpoint devices (hosts and storage) to be interconnected with 10 Gb/s Ethernet. For more information on FCoE, see the Fibre Channel Industry Association website at www.fibrechannel.org.

Conclusion
Ethernet storage fabrics are simple to integrate and can grow in any environment. For applications such as server consolidation and network consolidation, 10 GbE is ideal. Due to the ubiquitous nature of Ethernet and the ease of installation and management, Ethernet offers a combination of affordability, performance, and functionality. Converging the storage fabric onto Ethernet to support both file and block LAN traffic further enhances the value of Ethernet as the storage infrastructure.

For more information about the IP Storage Forum, please visit www.snia.org/ipstorage.