High-Speed Networking: A Systematic Approach to High-Bandwidth Low-Latency Communication / Edition 1

by James P. G. Sterbenz, Joseph D. Touch

Overview

Leading authorities deliver the commandments for designing high-speed networks
There are no end of books touting the virtues of one or another high-speed networking technology, but until now, there were none offering networking professionals a framework for choosing and integrating the best ones for their organization's networking needs. Written by two world-renowned experts in the field of high-speed network design, this book outlines a total strategy for designing high-bandwidth, low-latency systems. Using real-world implementation examples to illustrate their points, the authors cover all aspects of network design, including network components, network architectures, topologies, protocols, application interactions, and more.

Product Details

ISBN-13:	9780471330363
Publisher:	Wiley
Publication date:	05/02/2001
Series:	Networking Council , #5
Pages:	624
Product dimensions:	7.70(w) x 9.59(h) x 1.59(d)

About the Author

JAMES P. G. STERBENZ is Senior Network Scientist and Manager at BBN Technologies. Involved in high-speed technology research and development for many years, he has held leadership positions as chair of IEEE Communications Society Technical Committee on Gigabit Networking and the IFI Protocols for High-Speed Networks international steering committee.
JOSEPH D. TOUCH is Director of the Postel Center for Experimental Networking in the Computer Networks Division of Information Sciences Institute and Research Assistant Professor at USC. He was co-chair of the International Workshop on Protocols for High-Speed Networks.

Read an Excerpt

Introduction

The discipline of high-speed networking came into its own in the mid-1980s with intense research, first in high-speed switching technologies, and then on how to eliminate end-system bottlenecks. In the late 1990s, high-speed networking became so pervasive that just about everything involved high-speed networking, and little was only about high-speed networking. The field is now mature enough that we can consider the complete range of issues needed to deliver high performance to distributed communicating applications.

This book provides a systemic and systematic description of the principles of high-performance network architecture and design--systemic because we are concerned with the performance of the entire networked system or systems and systematic because we will consider in turn all the factors that are required for high speed networking:

• Network subsystems (such as routers, switches, and end systems).
• The protocols that glue these subsystems together.
• The entire network as a complex system of systems.
• The end-to-end delivery of high-bandwidth, low-latency data to applications.
• Applications that use and drive the demand for high-speed networks.

This is not a book about any particular technology--but where appropriate, we will provide illustrative examples of particular aspects of particular technologies, ideas from the research community, and historical foundations. We will not assume that networking began with the Internet, and operating systems began with Unix.

1.1 Bandwidth and Latency

We define high-performance networking as having twofundamental characteristics: high bandwidth and low latency. The motivation for high-performance networks is to support high-performance distributed applications, including remote access by a user to an application located elsewhere. The key performance metric might be surprising until some thought is given to the matter: delay. If the total delay in transferring chunks of data between applications is zero, then there is no difference whether the applications and users are distributed throughout a room or across the world. Note that we have not yet said anything about bandwidth.

1.1.1 Individual Delay

There are two components to this delay. The first is the time it takes to transmit a chunk of data out of the sending application, and the second is the time it takes to propagate across the network to the receiving application. If the chunk is only 1 bit long, then bandwidth doesn't matter very much; a 9600-b/ s telephone connection will only impose a chunk transmission delay of 104 s. Thus, the driver for high bandwidth between applications is the need to transfer large chunks of data with low delay. When applications are interactive, that is, in response to an action taken by the user, we want the request-response loop to complete quickly. Experience has shown that subsecond response time is necessary for reasonable productivity with interactive applications, and approaching 100-ms response is desirable. For example, if we are requesting a 1-KB Web page, we need a data rate of 100 kb/ s to transmit the page quickly enough. If the page contains 1-MB high-resolution images, we need a data rate of 100 Mb/ s. Therefore, low interapplication delay motivates the need for high-bandwidth paths between the applications.

Equally important is the delay component for propagation through the network, end systems, and application programs where the data is stored and accessed. It is of little use to the end user if data is being transferred at 100-Mb/s rates along network links, but is suffering delays of seconds in servers, network nodes, or at the end systems. Similarly, the latency of control messages, supporting session or connection control, can be critical to application performance. In this case, we are concerned not only with the latency of a single end-to-end trip but also with minimizing the number of round-trip delays on which the application must wait.

Note that some applications, such as remote backup, are generally considered to require high bandwidth, but are not particularly sensitive to latency. This is still driven by the application delay, but it is the need to transmit terabytes or petabytes of data in a time frame of minutes to hours, rather than days to weeks. Some applications, such as real-time, two-way voice conversations and process control, need low latency alone.

Finally, some applications, such as interactive Web browsing with 100-ms response, require both high bandwidth and low latency. We will be most concerned with networks and applications that require both high bandwidth and low latency.

1.1.2 Aggregate Bandwidth

There are two ways of looking at bandwidth: on a per link or stream basis and in aggregate (with respect to a particular component, subnetwork, or application). We are concerned with both. We have thus far motivated the former, since designing systems to provide individual high-bandwidth, low-latency paths directly benefits individual applications. We are also concerned with the scalability of high-performance networks. Therefore, how to build large routers and switches and how to scale high-performance networks will also be a focus of this book. This requires that individual components such as switch elements be scalable and that signaling and routing algorithms are designed to avoid explosion in state and control messages. Hierarchy is an important way to help with this problem. Giganode networks are of concern in conventional networking among personal and enterprise computers. Teranode and petanode networks will be of concern as we move to ubiquitous computing and wireless networking, where an individual, residence, or office may each have tens to thousands of networked smart devices.

1.2 What Is High Speed?

In this book, we have generally avoided the use of a specific prefix multiplier such as gigabit, terabit, petabit, or exabit. This is for two reasons. First, this is a book of architectural and design principles, not a snapshot of particular time frames or technologies. While the science of high-performance networking is still evolving, it is mature enough to codify principles that will continue to be valid as we move from gigabit to petabit networking and beyond. Second, the effective rate at which network components operate decreases as we move up the protocol stack and from the network core out to the end system. There are two reasons for this. The need for the network to aggregate vast numbers of high-performance interapplication flows dictates that the core of the network must be higher capacity than the end systems. Furthermore, it is easier to design components for high-speed communications whose sole purpose is networking than it is to optimize end systems with multiple roles and applications.

In the late 1990s, deployed link layers and multiplexors, such as Sychronous Optical Network (SONET), operated on the order of tens of gigabits per second, switches and routers at several gigabits per second per link, end system host-network interfaces in the range of 10 to 100 Mb/ s, and applications typically on the order of several megabits per second. By the early 2000s, link bandwidth and switch bandwidth had increased an order of magnitude or two, but local access and end-system bandwidth continued to lag. The challenge of designing end-to-end systems that deliver high-bandwidth and low-latency performance to the application is one of the focuses of this book. We will consider techniques such as zero-copy transfer using memory mapping [Sterbenz 1990a] and masking the speed-of-light delay between applications [Touch 1989]. This requires a high-speed packet-switched [Turner 1986a] or optical infrastructure between the end systems.

1.2.1 High-Speed Technologies

Many of the principles described in this book are relatively independent of the core technology. However, in some cases, the choice among conventional electro-optical, all-optical, wireless, and satellite network subsystems can have profound impact in the network. Examples include the high latency of geosynchronous satellite links, long burst errors of wireless channel fades, and the inability for all-optical paths to buffer data. These aspects will receive specific attention as appropriate.

We will also consider the performance implications of active networking, which consists of introducing programmability and functionality into network nodes to provide enhanced services traditionally located in end systems.

1.2.2 Barriers and Constraints

There are several significant barriers to the deployment of high-performance networks, in addition to the specific problems of designing end-to-end systems to be high bandwidth and low latency.

1.2.2.1 Bandwidth--Delay Product

Designing systems with high bandwidth, particularly over wide area networks, provides a serious challenge. While we can design individual components and subsystems for high-performance, we cannot decrease the latency due to the speed-of-light signal propagation over distance. Thus, there will always be a latency component due to the speed of light, which is significant to end-to-end performance in wide area networks.

The bandwidth--delay product refers to the amount of data in transit (r bit/ s d s rd bits). This has significant impact on the design of network components since rd bits of data must be handled in flight, and perhaps buffered in network components. The problem worsens as improving technology increases the bandwidth of optical links. As we begin to consider how to extend the Internet to interplanetary scales for the Mars and future Jupiter space missions [Cerf 1998], the prospect of petabits or exabits in flight is staggering. Furthermore, the design of protocols and applications for high-bandwidth--delay product is difficult, since over the interval that an application or protocol attempts to exert control, a large amount of data (2rd) may have flowed. By the time the control mechanisms take effect, the conditions that triggered them may no longer exist. Thus, when considering end-to-end protocols and applications, we will be just as concerned with high bandwidth--delay product networks as with bandwidth or latency alone.

1.2.2.2 Legacy and Heterogeneity

Two other barriers to the deployment of high-performance networks are of a more practical nature: legacy systems and heterogeneity. Although we might like to use the principles espoused in this book to start from scratch and design perfect high-performance networks, end systems, protocols, and application interfaces, this is clearly not practical. The world is full of legacy systems, and decades of experience have shown that it is very difficult to replace systems with a large installed base, particularly on which significant amounts of existing protocols and software rely. Even relatively minor changes like the introduction of multicast in the network and the transition from IPv4 to IPv6 can be quite painful. Proposals as drastic as the complete replacement of IP by ATM in the Internet are dead on arrival. Thus, this book will consider not only how to design new systems, protocols, and applications, but also how to use high-performance design principles to optimize existing network infrastructure and protocols, such as the Internet, Web, and TCP-based applications, with minimum impact.

A related barrier is that of heterogeneous applications and end systems, and results from several factors: Technological progress results in heterogeneous systems, even if only during a transition period (which is typically far longer than hoped). Heterogeneity also results from a desired variety in end-system applications and operating systems. Finally, heterogeneity results in the optimization of protocols and data formats to particular applications. As an example, the overhead in transcoding gateways in the network could be eliminated by using common data formats for all multimedia applications, but perhaps at a significant penalty in the end system and application itself, with considerable loss of flexibility to the end user.

1.3 Organization of This Book

Chapter 2 introduces the axioms and principles that guide the research, architecture, design, and deployment of high-speed networks. Chapters 3 and 4 then look at the network architecture as a whole. Chapter 3 considers high-speed network architecture, in particular topology, geography, and the location of functionality based on resource tradeoffs. Chapter 4 discusses network control and signaling. Chapters 5 and 6 consider how to design high-speed network components (links, switches, and routers) and end systems, respectively. Chapter 7 considers the end-to-end issues and transport protocols that interface to applications. Chapter 8 discusses high-performance applications and their interaction with the protocols. Finally, Chapter 9 looks ahead at the future of high-performance networking, as technologies and the contexts in which they are deployed change in foreseeable and unforeseeable ways. Chapters 3 through 8 begin with the canonical system and protocol diagram shown in Figure 1.1, with the focus of each chapter highlighted.

Chapter 1--Introduction

Chapter 1 is a good place to state the sole criterion for deciding what topics to include in the rest of the chapters: relevance to high-speed networking. Very few books with the terms "high-speed" or "gigabit" in their titles are actually about the issues and principles of high-speed networking. Rather, they are about particular networking technologies [such as asynchronous transfer mode (ATM) or Gigabit Ethernet] or classes of applications (such as multi-media). Similarly, there are entire topics that we will only touch on briefly as they relate to high-speed networking, and resist the temptation to present more thorough coverage. Examples include quality of service, routing, and traffic management.

Illustrating with a common misconception, although a number of design decisions were made for ATM (allegedly for high-performance implementation), there is nothing inherently high performance about the ATM architecture and protocol suite as a whole. While we will discuss some of the important aspects (such as fast packet switching), we will use other aspects as a counter-example of how not to design high-performance networks (for example, small cell size and complexity of signaling and traffic management). Similarly, there is nothing inherently high bandwidth about multimedia applications. The world is full of useful low-bandwidth video streaming, although interactive multimedia applications require low latency. Interactive data applications, such as Web browsing with high-resolution images and 100-ms response time, are far more bandwidth-intensive and latency-sensitive than video streaming.

Chapter 2--Fundamentals and Design Principles

Chapter 2 is the core foundation for the rest of the book, providing a set of axioms to guide our understanding of high-speed network architecture and design. Section 2.1 provides a brief history of networking. Section 2.2 discusses interapplication delay as the driver for high-speed networking, introduces an ideal zero-latency, infinite-bandwidth network model, and considers the barriers to deploying such a model. The axioms that will be described are as follows:

Ø. Know the Past, Present, and Future. Genuinely new ideas are extremely rare. Almost every "new" idea has a past full of lessons that can either be learned or ignored. "Old" ideas look different in the present because the context in which they have reappeared is different. Understanding the difference tells us which lessons to learn from the past and which to ignore. The future hasn't happened yet, and is guaranteed to contain at least one completely unexpected discovery that changes everything. On the other hand, we can prepare for the future by constantly reevaluating tradeoffs in the face of emerging technology, and questioning the basic assumptions of the present.

I. Application Primacy. The sole and entire point of building a high-performance network infrastructure is to support the distributed applications that need it. Interapplication delay drives the need for high-bandwidth, low-latency networks.

II. High-Performance Paths Goal. The network and end systems must provide a low-latency, high-bandwidth path between applications to support low interapplication delay. We introduce an ideal network, and consider path establishment, path protection, store-and-forward avoidance, blocking avoidance, efficient transfer of control, and information assurance.

III. Limiting Constraints. Real-world constraints make it difficult to provide high-performance paths to applications. These include the speed of light, limits on channel capacity and switching rate, heterogeneity, policy and administration, cost and feasibility, backward compatibility, and standards.

Section 2.3 presents the design principles and tradeoffs that form the basis of our approach for the research, architecture, design, and deployment of new communications and network technology.

IV. Systemic Optimization Principle. Networks are systems of systems with complex compositions and interactions at multiple levels of hardware and software. These pieces must be analyzed and optimized in concert with one another. To this end, we discuss the need to consider the side effects of optimizations and the benefits of simple, open, and flexible architectures.

The Systemic Optimization Principle is the umbrella for the design principles in the rest of the book:

1. Selective Optimization Principle. It is neither practical nor feasible to optimize everything. Spend implementation time and system cost on the most important contributors to performance.

2. Resource Tradeoff Principle. Networks are collections of resources. The relative composition of these resources must be balanced to optimize cost and performance.

3. End-to-End Arguments. Functions required by communicating applications can be correctly and completely implemented only with the knowledge and help of the applications themselves. Providing these functions as features within the network itself is not possible.

4. Protocol Layering Principle. Layering is a useful abstraction for thinking about networking system architecture and for organizing protocols based on network structure.

5. State Management Principle. The mechanisms for installation and management of state should be carefully chosen to balance fast, approximate, and coarse-grained against slow, accurate, and fine-grained.

6. Control Mechanism Latency Principle. Effective network control depends on the availability of accurate and current information. Control mechanisms must operate within convergence bounds that are matched to the rate of change in the network and latency bounds to provide low interapplication delay.

7. Distributed Data Principle. Distributed applications should select and organize the data they exchange to minimize the amount of data transferred and the latency of transfer and allow incremental processing of data.

8. Protocol Data Unit Principle. The size and structure of PDUs are critical to high-bandwidth, low-latency communication.

For each of these principles, several refinement principles and corollaries are discussed. For example, while the Protocol Layering Principle indicates the benefits of layered protocols, we will need the corollary that Layering as an Implementation Technique Performs Poorly. The rest of the chapters will present refinements and contextual versions of these principles as appropriate.

Section 2.5 introduces some of the most important design techniques that will be used to support the principles:

• Scaling time and space
• Masking the speed of light
• Specialized hardware implementation
• Parallelism and pipelining
• Data structure optimization
• Cut-through and remapping

Finally, Section 2.6 provides a summary, complete list of axioms and principles, and suggestions for further reading.

Chapter 3--Network Architecture and Topology

Chapter 3 considers the overall architecture of the network. Section 3.1 discusses the issues that are important in engineering an entire network. This consists of the topology and structure to provide high-bandwidth, low-latency paths through the network and the impact of geography. Section 3.2 discusses how to achieve large scale in networks and the use of hierarchy to manage complexity. Then, Section 3.3 examines the resource tradeoffs between bandwidth, memory, and processing, constrained by latency requirements. This is important in determining the proper location of functionality and services that the network should provide. Finally, Section 3.4 presents a summary and list of principles introduced throughout the chapter.

Chapter 4--Network Control and Signaling

Chapter 4 describes control issues in high-speed networks. Section 4.1 considers signaling and control issues, examining the overheads involved in the spectrum of connectionless to hard connection-oriented networks, and the mechanisms used in optical networks. Section 4.2 considers traffic management and congestion control in high-speed networks. Section 4.3 describes the impact of dynamic paths on high-speed networks. Section 4.4 touches on issues important in the monitoring and control of high-speed networks. Section 4.5 summarizes and presents the list of principles introduced in Chapter 4.

Chapter 5--Network Components

Chapter 5 examines the components that make up the network: links and nodes. Section 5.1 considers issues in the design and use of high-speed links of a variety of technologies. Section 5.2 describes the functional components needed in switches and routers. Traditional Internet router design is presented, and the performance bottlenecks are discussed. Then Section 5.3 considers how to design high-bandwidth, low-latency connection-oriented fast packet switches. Section 5.4 delves into the internal structure of the switch datapath, considering the design of buffering schemes and scalable switch fabrics that minimize blocking and utilize cut-through paths. Section 5.5 considers how to apply the techniques of high-speed fast packet switching design to high-speed connectionless datagram switches. In particular, techniques for fast address lookup, packet classification, and scheduling, will be discussed. Section 5.6 considers higher-layer protocol processing and the design of high-speed active network nodes. Finally, Section 5.7 presents a summary, suggestions for further reading, and list of principles.

Chapter 6--End Systems

In Chapter 6, the discussion moves to the edge of the network, and considers how end systems must be designed to allow the end-to-end, low-latency, high-bandwidth communications. The end system is particularly sensitive to an understanding of where bottlenecks really lie and the resultant systemic optimization [Clark 1990]. Section 6.1 describes the organization of the end system as a whole and introduces the hardware and software components of concern. It also discusses the traditional layered I/O-based protocol implementation, and presents the ideal end-system implementation model, based on the ideal network model from Chapter 2. Section 6.2 describes protocol software and the host operating system for high-speed networking. Principles and techniques, such as critical path optimization, integrated layer processing (ILP), minimizing operating system context switches, and exploiting the virtual memory subsystem, are discussed. Section 6.3 discusses host architecture and organization. Particular emphasis is placed on processor-memory interconnects to support a nonblocking, high bandwidth path between the network interface and application memory. Section 6.4 describes the implementation of high-speed host-network interfaces, and the partitioning of functionality between hardware and software. Finally, Section 6.5 summarizes, reiterates the design principles, and provides some pointers for further reading.

Chapter 7--End-to-End Protocols

Chapter 7 considers the end-to-end communication on which distributed applications rely. This functionality is provided by transport protocols, which are of central importance to the chapter. Section 7.1 introduces the end-to-end functionality and mechanisms provided by transport protocol, and presents the end-to-end arguments. The end-to-end arguments are the key determinant of end-to-end functionality. The important issue of open-versus closed-loop feedback control is presented. The remaining sections examine the different end-to-end mechanisms in more detail. Section 7.2 discusses connection control, and Section 7.3 describes framing and multiplexing. Section 7.4 describes error control mechanisms, and Section 7.5 presents flow and congestion control mechanisms. The use of open-loop control and decoupling error from and flow control are key principles. Section 7.6 considers high-performance end-to- end data integrity and security mechanisms. Section 7.7 provides the summary and suggestions for further reading.

Chapter 8--Networked Applications

Chapter 8 returns us to the motivation for high-speed networks: high-performance applications and the interface to the protocols and network. Section 8.1 discusses the characteristics of high-speed applications and the criteria we will use to determine whether an application should be considered "high-speed." We classify applications as real-time, interactive, or deadline constrained. Section 8.2 describes applications by class: information access, telepresence, distributed computing, or a composition thereof, and relates them to the characteristics of the previous section. Section 8.3 explores how applications can adapt to less than desired network bandwidth and latency and, in particular, how to mask the effects of the speed-of-light delay. Section 8.4 describes the interaction of the application and the protocols and network, in particular application feedback and control mechanisms. Section 8.5 summarizes the chapter and principles.

Chapter 9--Future Directions and Conclusion

Chapter 9 concludes the book. Section 9.1 looks toward the future, with a discussion on how the relative changes in resource tradeoffs can fundamentally alter the ways in which networks and protocols are designed. Section 9.2 considers some of the technology and applications that show promise to enable and challenge high-speed networking in the future. Finally, Section 9.3 provides a brief conclusion to the book.

1.4 Who Should Read This Book

The author of this book assumes that the reader understands computer networks, computer architecture, and operating systems at the level that would be achieved by completing a rigorous undergraduate computer science or computer engineering curriculum. This book is intended for graduate students and professionals who wish to understand the principles of high-performance network design, and serves as a starting point to the vast body of literature describing original research on the subject. This book is particularly appropriate for use in a special-topics graduate course in high-speed networking.

While targeted to high-speed network design, we believe that the unique approach of stating axioms and design principles up front and then providing contextual refinements will provide insight into system architecture and design that transcends high-speed networking. This book provides examples of particular network and protocol technologies to illustrate high-speed network design principles, but it is not a comprehensive treatment of any particular technology.

Table of Contents

Networkng Council Foreword.

Acknowledgments.

Introduction.

Fundamentals and Design Principles.

Network Architecture and Topology.

Network Control and Signaling.

Network Components.

End Systems.

End-to-End Protocols.

Networked and Applications.

Future Directions and Conclusion.

References.

Appendix A: Axioms and Principles.

Appendix B: Acronyms.

Index.

Editorial Reviews

Having decided that high-speed networking is mature enough to consider the complete range of issued needed to deliver high performance to distributed communicating applications, network specialists Sterbenz and Touch offer a systemic and systematic description of the principles of high-performance network architecture and design<-->systemic because they are concerned with the performance of the entire networked system or systems, and systematic because they consider in turn all the factors that are required. Annotation c. Book News, Inc., Portland, OR (booknews.com)

Booknews