Agile Software Development Ecosystems

Paperback (Print)
Buy New
Buy New from BN.com
$44.81
Used and New from Other Sellers
Used and New from Other Sellers
from $1.99
Usually ships in 1-2 business days
(Save 96%)
Other sellers (Paperback)
  • All (13) from $1.99   
  • New (8) from $23.75   
  • Used (5) from $1.99   

Overview

In a highly volatile software development environment, developers must be nimble, responsive, and able to hit a moving target--in short, they must be agile. Agile software development is designed to address this need for speed and flexibility. Agility describes a holistic, collaborative environment in which you can both create and respond to change by focusing on adaptability over predictability, people over process. Agile software development incorporates proven software engineering techniques, but without the overhead and restrictions of traditional development methodologies. Above all, it fulfills its promise of delivering software that serves the client's business needs.

Written by one of the leaders of the Agile movement, and including interviews with Agile gurus Kent Beck, Robert Charette, Alistair Cockburn, Martin Fowler, Ken Schwaber, and Ward Cunningham, Agile Software Development Ecosystems crystallizes the current understanding of this flexible and highly successful approach to software development. It presents the key practices of all Agile development approaches, offers overviews of specific techniques, and shows how you can choose the approach that best suits your organization.

This book describes--in depth--the most important principles of Agile development: delivering value to the customer, focusing on individual developers and their skills, collaboration, an emphasis on producing working software, the critical contribution of technical excellence, and a willingness to change course when demands shift. All major Agile methods are presented:

  • Scrum
  • Dynamic Systems Development Method
  • Crystal Methods
  • Feature-Driven Development
  • Lean Development
  • Extreme Programming
  • Adaptive Software Development

Throughout the book, case stories are used to illustrate how Agile practices empower success around the world in today's chaotic software development industry. Agile Software Development Ecosystems also examines how to determine your organization's Agile readiness, how to design a custom Agile methodology, and how to transform your company into a truly Agile organization.

0201760436B03042002

Read More Show Less

Editorial Reviews

From Barnes & Noble
The Barnes & Noble Review
It's no wonder "agile" software development methods are rapidly gaining popularity: they promise developers more respect and less bureaucracy, more speed and less aggravation, a greater likelihood of project success, and less chance of going stark raving bonkers along the way. But which agile methodology (if any) is right for your organization?

Jim Highsmith knows all seven leading approaches like the back of his hand. In Agile Software Development Ecosystems, he compares all seven, helping you customize the right approach to your unique requirements. Drawing upon interviews with the creators of each methodology, he illuminates Scrum, the Dynamic Systems Development Method, CrystalMethods, Feature-Driven Development, Lean Development, his own Adaptive Software Development (ASD), and the best-known of them all, Kent Beck's eXtreme Programming.

While there are significant differences among these methodologies, you shouldn't underestimate the challenge of implementing any of them in the traditional Dilbert-like software organization. For agility to work, you need more than a methodology, you need an "ecosystem" that supports it.

Highsmith says agile "ecosystems" need to encompass three elements: collaborative values and principles, a methodology that's as light as possible, and a "chaordic" perspective that respects the fact that real-world organizations exhibit both chaos and order and can't be managed solely through conventional project management and development life-cycle practices. Along the way, he uses multiple case studies to illuminate what it takes to make each agile methodology work -- and to offer practical help for folks who want to nudge their organization toward agility in any form. (Bill Camarda)

Bill Camarda is a consultant, writer, and web/multimedia content developer with nearly 20 years' experience in helping technology companies deploy and market advanced software, computing, and networking products and services. His 15 books include Special Edition Using Word 2000 and Upgrading & Fixing Networks For Dummies®, Second Edition.

Read More Show Less

Product Details

  • ISBN-13: 9780201760439
  • Publisher: Addison-Wesley
  • Publication date: 3/28/2002
  • Series: Agile Software Development Series
  • Edition description: New Edition
  • Pages: 448
  • Product dimensions: 7.32 (w) x 8.97 (h) x 1.18 (d)

Meet the Author

Jim Highsmith is a well-known consultant, software developer, writer, and speaker. He is a founding member of the AgileAlliance, serving on its first board, and is coauthor of the Agile Manifesto. Jim is director of the Agile Project Management Advisory Service for the Cutter Consortium. He is also the author of Adaptive Software Development (Dorset House), winner of the 2000 Jolt Award.

0201760436AB03112002

Read More Show Less

Read an Excerpt

From February 11 to 13, 2001, at the Lodge at Snowbird ski resort in the Wasatch Mountains of Utah, 17 people met to talk, ski, relax, and try to find common ground. What emerged was the Agile Software Development movement. Representatives from Extreme Programming (XP), Scrum, the Dynamic Systems Development Method (DSDM), Adaptive Software Development (ASD), Crystal Methods, Feature-Driven Development (FDD), Pragmatic Programming, and others sympathetic to the need for an alternative to document-driven, rigorous software development processes convened. What this meeting produced—The Manifesto for Agile Software Development, signed by all 17 of the participants—was symbolic. It declares that:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan.

That is, while there is value in the items on the right, we value the items on the left more.

This value statement has a number of fascinating aspects, not the least of which was getting 17 people to agree to it. Although this was a group of experienced and recognized software development "gurus," the word uncovering was selected to indicate that the authors don't have all the answers and don't subscribe to the silver-bullet theory.

The phrase "by doing it" indicates that the authors actually practice these methods in their own work. Ken Schwaber told the story of his days of selling tools to automate comprehensive, "heavy" methodologies. Impressed by the responsiveness of Ken's company, Jeff Sutherland asked him which of these rigorous methodologies he used for internal development. "I still remember the look on Jeff's face," Ken remarked, "when I told him, ‘None. If we used any of them, we'd be out of business.'" The authors want to help others with Agile practices and to further our own knowledge by learning from those we try to help.

The value statements have a form. In each statement, the first segment indicates a preference, while the latter segment describes an item that, while important, is of lesser priority. This distinction lies at the heart of agility, but simply asking people to list what's valuable doesn't flesh out essential differences. Roy Singham, CEO of ThoughtWorks, put it well when he said that it's the edge cases, the hard choices, that interest him. "Yes, we value planning, comprehensive documentation, processes, and tools. That's easy to say. The hard thing is to ask, ‘What do you value more?'" When push comes to shove—and it usually does—something must give, and we need to be clear about what stays and what gives.

The first value statement recognizes the importance of processes and tools but stresses that the interaction of talented individuals is of far more importance. Rigorous methodologies and their business process reengineering brethren place more emphasis on process than people. Agile development reverses this trend. If individuals are unique, and we believe they are, then processes should be melded to individuals and teams, not the other way around.

Similarly, while comprehensive documentation is not necessarily harmful, the primary focus must remain on the final product—working software. This means that every project team needs to determine, for itself, what documentation is absolutely essential to delivering working software. Working software tells the developers and sponsors what they really have in front of them—as opposed to promises of what they might have in front of them. Working software can be shipped, modified, or scrapped, but it is always real.

Contract negotiation, whether through an internal project charter or an external legal contract, isn't a bad practice, just a seriously insufficient one. Customers want a product that conforms to their needs—at the time of delivery. "Contract negotiation encourages the addition of buffers contingency," says colleague Ron Holliday. "This makes projects take even longer, drives up costs, and reduces responsiveness to change. A collaborative arrangement is a team effort between customer and supplier to deliver the best possible solution." Contracts and project charters provide boundary conditions within which the parties can work, but only through ongoing collaboration can a development team hope to understand and deliver what the client wants.

No one can argue that following a plan is a good idea—right? Well, yes and no. In a world of business and technology turbulence, scrupulously following a plan can have dire consequences, even if it's executed faithfully. However carefully a plan is crafted, it becomes dangerous if it blinds you to change. As you will see in several of the case studies presented in this book, few, if any, of the projects delivered what was planned for in the beginning. And yet they were successful because the development team was Agile enough to respond again and again to external changes. In the Information Age, planning is important, but accepting that deviations from any plan are "normal" is even more important.

The meeting at Snowbird was incubated at a spring 2000 gathering of Extreme Programming leaders organized by Kent Beck. At that meeting in Oregon, a few "outsiders" but "sympathizers," such as myself, were invited, and attendees voiced support for a variety of "light" methodologies. During 2000, a number of articles referenced the category of "light" or "lightweight" practices. In conversation, the soon to be "Agile" authors didn't like the moniker "light," but it stuck at that time.

In September 2000, "Uncle Bob" Martin of Object Mentor in Chicago started what was to become the Agile ball rolling with an email: "I'd like to convene a short (two-day) conference in the January to February 2001 time frame here in Chicago. The purpose of this conference is to get all the lightweight method leaders in one room. All of you are invited, and I'd be interested to know who else I should approach." Bob set up an online group site, and the discussions raged. Early on, Alistair Cockburn weighed in with an epistle that identified the general disgruntlement with the word "light": "I don't mind the methodology being called light in weight, but I'm not sure I want to be referred to as a lightweight attending a lightweight methodologists meeting. It somehow sounds like a bunch of skinny, feeble-minded lightweight people trying to remember what day it is."

The fiercest debate was over location! There was serious concern about Chicago in wintertime—cold and nothing fun to do. Someone suggested Snowbird, Utah—cold, but fun things to do, at least for those who ski on their heads, as Martin Fowler tried on the first day. Someone else mentioned Anguilla in the Caribbean—warm and fun, but time consuming to get to. In the end, Snowbird and skiing won out.

The Agile Manifesto value statements represent a deeper theme that drives many, but not all, of the Manifesto's authors. At the close of the two-day meeting, Bob Martin joked that he was about to make a "mushy" statement. Although tinged with humor, no one disagreed with Bob's sentiments—that we all felt privileged to work with a group of people who held a set of compatible values, based on trust and respect for each other, promotion of organizational models based on people, and the desire to build collaborative communities in which to work. At the core, I believe Agile developers are really about mushy stuff—about delivering products to customers while operating in a vibrant, sustaining workplace. So in the final analysis, the meteoric rise of interest in—and criticism of—Agile Software Development is about the mushy stuff of values and culture.

For example, I think that, ultimately, XP has mushroomed in use and interest not because of pair programming or refactoring but because, taken as a whole, the practices define a developer community freed from the baggage of Dilbertesque corporations. Kent Beck tells the story of an early job in which he estimated a programming effort to be six weeks for two people. After his manager reassigned the other programmer at the beginning of the project (effectively halving the resources), Kent completed the project in twelve weeks—and felt terrible! The boss, of course, harangued Kent throughout the second six weeks. Somewhat despondent because he was such a "failure" as a programmer, Kent finally realized that his original estimate of six weeks was extremely accurate—for two people—and that his "failure" was really his manager's failure to take the responsibility for removing project resources. This type of situation goes on every day with individuals who don't want to make hard tradeoff decisions, so they impose irrational demands.

The Agile movement is not anti-methodology; in fact, we seek to restore credibility to the concept of methodology. We want to restore a balance. We accept modeling, but not in order to file some diagram in a dusty corporate repository. We accept documentation, but not hundreds of pages of never-maintained and rarely used tomes. We plan, but recognize the limits of planning in a turbulent environment. Those who would brand proponents of FDD or Scrum or any of the other Agile approaches as "hackers" are ignorant of both the approaches and the original definition of the term hacker—a programmer who enjoys solving complex programming problems, rather than one who practices either ad hoc development or "cracking." Agile Software Development incorporates proven software engineering techniques, but without the overhead of traditional methodologies.

The Agile Alliance was born in early 2001, but the history of the various approaches and the people who developed them goes back 10 to 15 years. This book describes these practices and the principles behind them, but more important, it delves into the people—the people who are developing the practices and the people who use them to deliver business value to their customers.

Finding a Balance

The left side of the Agile Manifesto value statements indicates what Agilists consider more important than the items on the right side. This should not be construed as indicating that tools, process, documentation, or contracts are unimportant. There is a tremendous difference between one thing being more or less important than another and being unimportant. Judicious use of tools is absolutely critical to speeding software development and reducing its costs. Contracts are one vital component to understanding developer-customer relationships. Documentation, in moderation, aids communication, enhances knowledge transfer, preserves historical information, and fulfills governmental and legal requirements.

But Agilists take a certain perspective on these topics. Try this exercise. For each of the Manifesto value statements, construct two questions along the lines of the following:

  • Could we have a successful project by delivering documentation without working software?
  • Could we have a successful project by delivering working software without any documentation?

By looking at these two end points, we can better grasp relative importance. Although there has to be a balance—documentation and working software, contracts and collaboration, responsiveness and planning, people and process—we have to delineate the extremes, the end points, so that organizations, teams, and individuals can find their own balance points. If we start out trying to find the middle ground, we never will.

One of the great contributions of XP's "extremoes"—Kent Beck, Ron Jeffries, and Ward Cunningham—is that they have staked out positions that have stirred debate in ways that taking moderate positions never would. When Ron says, "Great designs emerge from zero anticipatory design and continuous refactoring," he is challenging himself, and us, to rethink our assumptions about software development. We have to understand the limits before we can understand the balance points.

So although I realize the value of documentation, contracts, plans, and processes, there are numerous sources of material about these subjects. My intention is to identify and define Agile Software Development, to articulate its practices and principles, so you can make your own decision about where on the spectrum you, or your organization, need to be.

Fundamental Questions

There are three fundamental questions that this book answers: (1) What kinds of problems does agility solve best? (2) What is agility? and (3) What are Agile Software Development Ecosystems (ASDEs)?

What Kinds of Problems Does Agility Solve Best?

Problems characterized by change, speed, and turbulence are best solved by agility. Some people argue that good practices are good practices (pair programming, customer focus groups, or feature planning, for example) and therefore ASDEs should not be "limited" to a particular problem type. While true in part, the question asks what problems Agile practices best solve, not just solve. So, while XP, Crystal, or Scrum can surely be used for a wide range of projects, they are particularly relevant to extreme or complex projects—those that have an accelerated time schedule combined with significant risk and uncertainty that generate constant change during the project.

As the level of change caused by rapidly evolving technology, business models, and products increases and the need for delivery speed accelerates, ASDEs' effectiveness increases quickly over rigorous methodologies.

What Is Agility?

Agility, like any other complex concept, has a number of definitions, but for me, the most clearly focused definition is

Agility is the ability to both create and respond to change in order to profit in a turbulent business environment.

Rather than shrink from change, Agile organizations harness or embrace change by being better than competitors at responding to changing conditions and by creating change that competitors can't respond to adequately. However, companies must determine what level of agility they require to remain competitive. Agility is only an advantage relative to competitors—a copper mining company doesn't need to be as agile as a biotechnology firm.

Other aspects of agility are also important: nimbleness or flexibility on the one hand, and balance on the other. Agile organizations are nimble (able to change directions quickly) and flexible (able to see how things that worked for them last week may not work as well next week). An Agile organization also knows how to balance structure and flexibility. If everything changes all the time, forward motion becomes problematic. Agile organizations understand that balancing on the edge between order and chaos determines success.

What Are Agile Software Development Ecosystems?

I began writing this book about Agile Software Development methodologies, but I kept worrying about the word "methodology" because it didn't fit with the focal points of Agile development—people, relationships, and uncertainty. Furthermore, by using the word "methodology," Agile practices are instantly compared to traditional software development methodologies—thereby using the wrong measuring stick for comparison. So I use the term "Agile Software Development Ecosystem" to describe a holistic environment that includes three interwoven components—a "chaordic" perspective, collaborative values and principles, and a barely sufficient methodology—and the term Agilists to identify those who are proponents of ASDEs.

Some people think that "Agile" means fewer processes, less ceremony, and briefer documents, but it has a much broader perspective, which is the primary reason for using the word ecosystem rather than methodology. Although fewer processes and less formality might lower development costs, they are not enough to produce agility. Focusing on people and their interactions and giving individuals the power to make quick decisions and to self-adapt their own processes are key to Agile ecosystems.

The American Heritage Dictionary defines ecosystem as "organisms and their environment: a localized group of interdependent organisms together with the environment that they inhabit and depend on." The Oxford English Dictionary extends this definition to include a constant interchange within the system, including both organic and inorganic elements. The word methodology conjures up a vision of things—activities, processes, tools. These are not inconsequential elements, but incomplete ones. The word ecosystem conjures up a vision of living things and their interactions with each other. Within an organizational context, an ecosystem can then be thought of as a dynamic, ever-changing environment in which people and organizations constantly initiate actions and respond to each other's actions. The word ecosystem focuses us on the dynamic interactions of individuals and teams rather than on the static lines on organization charts.

A Chaordic Perspective

To fully understand ASDEs, we need to understand each of the three components and how they relate to each other. First, Agilists share a view that organizations are chaordic—that every organization exhibits properties of both chaos and order that defy management through the use of linear, predictive planning and execution practices. Viewing organizations as chaordic means understanding that the predictability upon which traditional project management and development life cycle practices are built is a root cause of dysfunctionality between customer, management, and development organizations. A chaordic perspective impacts both how we respond to change and how we manage project teams and organizations.

In day-to-day project work, a chaordic perspective creates two outcomes that are 180 degrees out of sync with rigorous methodologies.

  • Product goals are achievable, but they are not predictable.
  • Processes can aid consistency, but they are not repeatable.

Although ASDEs involve careful planning, the fundamental assumption remains that plans, in a turbulent environment, are not predictable, at least at the level of project scope, schedule, and cost. Plans are hypotheses to be tested rather than predictions to be realized. However, the product goals of the business are achievable, in large part because Agile people adapt. They can "adapt" to an articulated vision and a schedule, scope, or cost goal through tradeoffs in the other two dimensions. Second, while process can aid people in working together, in volatile environments the idea of driving out process variation through measurement and correction—statistical process control—becomes an unworkable hypothesis. Changes that are the result of knowledge gained during the project, knowledge not discernable early in the project, require processes that can respond to change, not ones that attempt to eliminate it.

As Martin Fowler points out, the two fundamental characteristics of ASDEs are their focus on adaptability rather than predictability and on people rather than process (Fowler 2000). As you read through this book, you will see just how fundamental—and challenging to the status quo—these two principles really are. Being Agile means accepting that outcomes are not predictable and that processes are not repeatable. It even implies that as process repeatability increases, project predictability decreases.

Peter Senge uses the term "mental model" to identify the perspective, set of assumptions, stories, and beliefs that each of us carries in our mind that provide a context for thinking (Senge 1990). In organizations, the collective set of mental models defines an overall cultural context. Companies that are heavily sales oriented differ from those that are heavily engineering oriented. Companies whose driving strategy is customer intimacy differ from those whose driving force is product innovation. Companies whose mental model includes linearity, cause and effect, hierarchy, predictability, and control will operate very differently from one whose mental model includes collaborative networks, emergence, decentralization of power, and acceptance of unpredictability. One is Newtonian, the other chaordic.

A chaordic perspective draws on a complex adaptive systems model in which decentralized, independent agents (each of us) interact in self-organizing ways, guided by a set of simple, generative rules that create complex, emergent results. This perspective is examined in detail in my book Adaptive Software Development (Highsmith 2000) and is explicitly embraced in the philosophies of Agilists Ken Schwaber, Bob Charette, and Kent Beck.

XP provides an example of how methodology and culture fit together. At one level, XP is defined by a system of practices—pair programming, test-first development, refactoring, coding standards. But the values and principles of XP define a collaborative culture—how developers work together with customers, how individuals interact and treat each other as human beings.

Collaborative Values and Principles

The second piece of the interconnected web that defines ASDEs is the statement of collaborative values and principles. While it is difficult to characterize the Agile Manifesto in one word, "collaborative" seems to be the best single adjective. Values and principles shape the ecosystem. Without a set of stated values and principles, an ecosystem is sterile, reflecting practices but not the people who interact within it.

A collaborative culture includes people and their relationships within a development team and with customers, management, and partnering teams within or external to their own company. Human dynamics, communications, and collaboration may be known as the "soft" sciences, but in practice, they may be the hardest to master. Principles and values help define a culture—the environment in which people want to work.

A Barely Sufficient Methodology

The final component of an ASDE is methodology. The traditional definition of methodology includes things such as roles, activities, processes, techniques, and tools. Alistair Cockburn summarizes these components when he defines methodology as "the conventions we agree to"—the ways in which people work together on a project. In The Social Life of Information, John Seely Brown and Paul Duguid (2000) discuss the major differences between process (as used by the business process reengineering movement) and practice. Processes are described in manuals; practices are what happen in reality. Process centrists relegate people to second place; practice centrists place people first. Process centrists focus on explicit (written down) knowledge, while practice centrists focus on tacit (internal) knowledge. The ASDE model provides a practice-centered, rather than a process-centered, approach to methodology.

There are two reasons to pursue barely sufficient methodologies: value and innovation. Streamlined methodologies concentrate on those activities that create value and ruthlessly eliminate non-value-adding activities. Programming usually adds value; process management often adds overhead. Bare sufficiency means keeping the former and eliminating the latter. Second, innovation and creativity flourish in chaordic environments, not orderly ones. Barely sufficient methodologies are cauldrons for breeding innovation.

Methodology also relates to organizational model. Agile methodologies contain minimal processes and documentation and reduced ceremony (formality). Agile methodologies are labeled "barely sufficient" (Cockburn) or "a little bit less than just enough" (Highsmith), or "minimal" (Charette). However, this streamlining of methodology isn't based just on reducing work effort but, more important, it is based on understanding the chaordic world view—one in which emergent (innovative) results are best generated at the "edge of chaos," perched midway between chaos and order.

Practices (or techniques) are the lifeblood of methodology. Whether it's pair programming, Scrum meetings, customer focus groups, or automated testing, the practices of ASDEs, carried out by talented and skilled individuals, produce results.

Changing Perspectives

In the software profession, we've used the words "methodology" and "process" for so long that they roll off the tongue and the pen without trouble. "Ecosystem" will take getting used to, but then, that's why I'm using the word—to foster a different perspective. "There is accumulating evidence that corporations fail because the prevailing thinking and language of management are too narrowly based on the prevailing thinking and language of economics," says Arie De Geus (1997). "They forget that their organizations' true nature is that of a community of humans."

To change thinking, we must change the language we use, so I use the word "ecosystem" to change our thinking about how software projects should be viewed. A chaordic perspective, a collaborative set of values and principles, and a barely sufficient methodology all combine and interact to form an Agile ecosystem. We cannot separate the three, at least in my mind, and I think most Agilists would agree. One could have a streamlined methodology but a linear, Newtonian view of organizations, and the result would not be Agile. One could have a streamlined methodology but a hierarchical, control-based work culture, and it would not be Agile. One could have a collaborative, people-oriented work culture but a rigid, predictive approach to planning and managing projects, and it would not be Agile. Agility requires all three.

The ultimate objective of this book is to describe new ways of working together to deliver business value to software customers. The heart of ASDEs is a core belief in people—their individuality and their interactions. It's impossible to discuss people and their ways of working together (ecosystem) without discussing values and principles. It's impossible to discuss values and principles without also discussing assumptions about how organizations do, or should, work. It's impossible to compare Agile approaches with non-Agile approaches using "methodology" as the only mechanism for comparison.

In closing, I need to state that I don't speak for anyone in the Agile community other than myself. I may interpret what Ken Schwaber says about Scrum, what Jeff De Luca says about FDD, or what Alistair Cockburn says about Crystal Methods, but they are my interpretations. In making generalizations about ASDEs, I'm sure I've made statements that would generate disagreement from 1 or more of the other 16 authors of the Agile Manifesto. However, I have talked to, corresponded with, or worked with all these authors, so although they are my own interpretations, they also reflect a deep sense of being part of, and contributor to, this community.

Although 17 individuals authored the Agile Manifesto, thousands support this effort. Many, many individuals have signed the Manifesto Web page, and an array of Web sites prominently display the statement "We support the Agile Manifesto." Agilists have stirred a healthy debate within the software development and project management communities. I hope this book will contribute to that debate and encourage others to join in it.

Jim Highsmith Salt Lake City, Utah

Read More Show Less

Table of Contents

Foreword.

Preface.

Finding a Balance.

Fundamental Questions.

What Kinds of Problems Does Agility Solve Best?

What Is Agility?

What Are Agile Software Development Ecosystems?

A Chaordic Perspective.

Collaborative Values and Principles.

A Barely Sufficient Methodology.

Changing Perspectives.

Introduction.

Book Organization and Conventions.

The Major Agile Ecosystems and Leaders.

Scrum.

Dynamic Systems Development Method (DSDM).

Crystal Methods.

Feature-Driven Development (FDD).

Lean Development (LD).

Extreme Programming (XP).

Adaptive Software Development (ASD).

Acknowledgments.

The Agile Software Development Series.

I. PROBLEMS AND SOLUTIONS.

1. The Change-Driven Economy.

Turbulence: Bubbles versus Trends.

Exploration versus Optimization.

Exploratory Projects.

Command-Control versus Leadership-Collaboration Cultures.

Thriving at the Edge.

2. IDX Systems Corporation.

The IDX Story.

An Agile Group in Action.

3. Agility.

Agility.

Creating and Responding to Change.

Nimbleness and Improvisation.

Conformance to Actual.

Balancing Flexibility and Structure.

“Agile” Studies.

Product Development in Internet Time.

“Heavy” Agile Projects.

Agile Software Development Ecosystems.

II. PRINCIPLES AND PEOPLE.

4. Kent Beck.

Reflections.

5. Deliver Something Useful.

HAHT Commerce, Inc.

Customer Delivery Principles.

Delivering Customer Value.

Voice of the Customer.

Working Software.

Frequent Delivery.

Work Together Daily.

Practices That Deliver Useful Features.

The Customer-Developer Interface.

Proxy Users.

Domain-Knowledgeable Developers.

Contracts: Shaping Customer Relationships.

Obviously It's Not Obvious.

6. Alistair Cockburn.

Reflections.

7. Rely on People.

ThoughtWorks.

Who Are You Calling Average?

Trust, Mistrust, and Communications.

Talent, Skill, and Process.

Process versus Skill.

Artifacts and Information Flow.

Innovation and Creativity.

The Fall and Resurrection of Programming.

Software through People.

8. Ken Schwaber.

Reflections.

9. Encourage Collaboration.

The Modern Transport Team at ITL.

A Cooperative Game of Invention and Communication.

Practice versus Process.

Documentation Is Not Understanding.

The Dimensions of Collaboration.

Real Teams.

10. Martin Fowler.

Reflections.

11. Technical Excellence.

The PDFS Team at Generali Group.

Agile Is Not Ad Hoc.

Removal of Defects.

Focus on Code.

Simple Design.

Big Bang versus Incremental.

Modeling and Abstraction.

Domain Recognition.

Documentation versus Conversation.

Specialists versus Generalists.

Quality versus Speed.

Establishment versus Anti-establishment.

Values and Principles.

Reflections.

12. Ward Cunningham.

Reflections.

13. Do the Simplest Thing Possible.

The Survey Controller Team at Trimble Navigation.

Musashi.

The Three Faces of Simplicity.

Simplicity as Minimalism.

Simplicity as Good Design.

Simplicity as Generative Rules.

Adapting Simple Rules.

A Final Note on Simplicity.

14. Jim Highsmith.

15. Be Adaptable.

The Mustang Team at Cellular, Inc.

The Great Divide: Predictability or Adaptability.

Our Changing Business Ecosystem.

Embracing Change.

Facilitate Change.

View Rework as a Virtue.

Control Final Components.

Constant Feedback at Multiple Levels.

Multiple Process Levels.

Balancing Adaptation with Anticipation.

Putting Lipstick on a Bulldog.

The Cost of Change.

Conform to Actual: Measuring Success.

Adaptability Is a Mindset.

16. Bob Charette.

Reflections.

III. AGILE SOFTWARE DEVELOPMENT ECOSYSTEMS.

17. Scrum.

The Scrum Process.

Pre-Sprint Planning.

Sprint.

Post-Sprint Meeting.

Monitoring Progress.

Scrum's Contributions.

18. Dynamic Systems Development Method.

Arie van Bennekum.

DSDM Principles.

The DSDM Process.

DSDM's Contributions.

19. Crystal Methods.

Methodology Design Principles.

The Crystal Framework.

Crystal Method Example: Crystal Clear.

Crystal's Contributions.

20. Feature-Driven Development.

The Singapore Project.

The FDD Process Model.

Process 1: Develop an Overall Model.

Process 2: Build a Features List.

Process 3: Plan by Feature.

Process 4: Design by Feature.

Process 5: Build by Feature.

Beyond the FDD Process Description.

Conceptual Similarities and Differences.

FDD's Contributions.

21. Lean Development.

EuroTel.

The Strategic Foundation of Lean Development.

Lean Development's Origins.

What Is Lean Development?

The Lean Development Environment.

Lean Development's Contributions.

22. Extreme Programming.

XP--The Basics.

XP Practices.

Values and Principles.

XP's Contributions.

23. Adaptive Software Development.

A Change-Oriented Life Cycle.

The Basic ASD Life Cycle.

Speculate: Initiation and Planning.

Collaborate: Concurrent Feature Development.

Learn: Quality Review.

Leadership-Collaboration Management.

ASD's Contributions.

IV. DEVELOPING AN ASDE.

24. Articulating Your Ecosystem.

Opportunity and Problem Domains.

Cultural Domain.

The Competence Culture.

The Control Culture.

The Collaboration Culture.

The Cultivation Culture.

Cultural Relativism.

Matching Methodology to Opportunity and Culture.

Methodology Selection.

Articulate Values and Principles.

25. Designing Your Agile Methodology.

Methodology Expectations.

Methodology Elements and the System of Practices.

Keep It Simple.

Practices and Principles

Methodology Design Principles.

Frameworks, Templates, and Scenarios.

Phase and Gate Life Cycle Frameworks.

Problem Domain Templates.

Scenarios.

Collaborative Methodology Design Steps.

Evaluate Project Objectives and Characteristics.

Design a Methodology Framework, Templates, and Scenarios.

Customize Templates to the Team.

A Customizing Approach.

Adapt the Template to Use.

Scaling.

Methodology Scaling: Balancing Optimizing and Adapting Elements.

Collaboration Scaling.

Architecture and Integration Scaling.

Agile Methodologies for the Enterprise.

26. The Agile Metamorphosis.

Chaordic Perspective.

Collaborative Values and Principles.

Barely Sufficient Methodology.

The Agility Ratings.

Final Thoughts.

Bibliography.

Index. 0201760436T04092002

Read More Show Less

Preface

From February 11 to 13, 2001, at the Lodge at Snowbird ski resort in the Wasatch Mountains of Utah, 17 people met to talk, ski, relax, and try to find common ground. What emerged was the Agile Software Development movement. Representatives from Extreme Programming (XP), Scrum, the Dynamic Systems Development Method (DSDM), Adaptive Software Development (ASD), Crystal Methods, Feature-Driven Development (FDD), Pragmatic Programming, and others sympathetic to the need for an alternative to document-driven, rigorous software development processes convened. What this meeting produced--The Manifesto for Agile Software Development, signed by all 17 of the participants--was symbolic. It declares that:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan.

That is, while there is value in the items on the right, we value the items on the left more.

This value statement has a number of fascinating aspects, not the least of which was getting 17 people to agree to it. Although this was a group of experienced and recognized software development "gurus," the word uncovering was selected to indicate that the authors don't have all the answers and don't subscribe to the silver-bullet theory.

The phrase "by doing it" indicates that the authors actually practice these methods in their own work. Ken Schwaber told the story of his days of selling tools to automate comprehensive, "heavy" methodologies. Impressed by the responsiveness of Ken's company, Jeff Sutherland asked him which of these rigorous methodologies he used for internal development. "I still remember the look on Jeff's face," Ken remarked, "when I told him, ‘None. If we used any of them, we'd be out of business.'" The authors want to help others with Agile practices and to further our own knowledge by learning from those we try to help.

The value statements have a form. In each statement, the first segment indicates a preference, while the latter segment describes an item that, while important, is of lesser priority. This distinction lies at the heart of agility, but simply asking people to list what's valuable doesn't flesh out essential differences. Roy Singham, CEO of ThoughtWorks, put it well when he said that it's the edge cases, the hard choices, that interest him. "Yes, we value planning, comprehensive documentation, processes, and tools. That's easy to say. The hard thing is to ask, ‘What do you value more?'" When push comes to shove--and it usually does--something must give, and we need to be clear about what stays and what gives.

The first value statement recognizes the importance of processes and tools but stresses that the interaction of talented individuals is of far more importance. Rigorous methodologies and their business process reengineering brethren place more emphasis on process than people. Agile development reverses this trend. If individuals are unique, and we believe they are, then processes should be melded to individuals and teams, not the other way around.

Similarly, while comprehensive documentation is not necessarily harmful, the primary focus must remain on the final product--working software. This means that every project team needs to determine, for itself, what documentation is absolutely essential to delivering working software. Working software tells the developers and sponsors what they really have in front of them--as opposed to promises of what they might have in front of them. Working software can be shipped, modified, or scrapped, but it is always real.

Contract negotiation, whether through an internal project charter or an external legal contract, isn't a bad practice, just a seriously insufficient one. Customers want a product that conforms to their needs--at the time of delivery. "Contract negotiation encourages the addition of buffers contingency," says colleague Ron Holliday. "This makes projects take even longer, drives up costs, and reduces responsiveness to change. A collaborative arrangement is a team effort between customer and supplier to deliver the best possible solution." Contracts and project charters provide boundary conditions within which the parties can work, but only through ongoing collaboration can a development team hope to understand and deliver what the client wants.

No one can argue that following a plan is a good idea--right? Well, yes and no. In a world of business and technology turbulence, scrupulously following a plan can have dire consequences, even if it's executed faithfully. However carefully a plan is crafted, it becomes dangerous if it blinds you to change. As you will see in several of the case studies presented in this book, few, if any, of the projects delivered what was planned for in the beginning. And yet they were successful because the development team was Agile enough to respond again and again to external changes. In the Information Age, planning is important, but accepting that deviations from any plan are "normal" is even more important.

The meeting at Snowbird was incubated at a spring 2000 gathering of Extreme Programming leaders organized by Kent Beck. At that meeting in Oregon, a few "outsiders" but "sympathizers," such as myself, were invited, and attendees voiced support for a variety of "light" methodologies. During 2000, a number of articles referenced the category of "light" or "lightweight" practices. In conversation, the soon to be "Agile" authors didn't like the moniker "light," but it stuck at that time.

In September 2000, "Uncle Bob" Martin of Object Mentor in Chicago started what was to become the Agile ball rolling with an email: "I'd like to convene a short (two-day) conference in the January to February 2001 time frame here in Chicago. The purpose of this conference is to get all the lightweight method leaders in one room. All of you are invited, and I'd be interested to know who else I should approach." Bob set up an online group site, and the discussions raged. Early on, Alistair Cockburn weighed in with an epistle that identified the general disgruntlement with the word "light": "I don't mind the methodology being called light in weight, but I'm not sure I want to be referred to as a lightweight attending a lightweight methodologists meeting. It somehow sounds like a bunch of skinny, feeble-minded lightweight people trying to remember what day it is."

The fiercest debate was over location! There was serious concern about Chicago in wintertime--cold and nothing fun to do. Someone suggested Snowbird, Utah--cold, but fun things to do, at least for those who ski on their heads, as Martin Fowler tried on the first day. Someone else mentioned Anguilla in the Caribbean--warm and fun, but time consuming to get to. In the end, Snowbird and skiing won out.

The Agile Manifesto value statements represent a deeper theme that drives many, but not all, of the Manifesto's authors. At the close of the two-day meeting, Bob Martin joked that he was about to make a "mushy" statement. Although tinged with humor, no one disagreed with Bob's sentiments--that we all felt privileged to work with a group of people who held a set of compatible values, based on trust and respect for each other, promotion of organizational models based on people, and the desire to build collaborative communities in which to work. At the core, I believe Agile developers are really about mushy stuff--about delivering products to customers while operating in a vibrant, sustaining workplace. So in the final analysis, the meteoric rise of interest in--and criticism of--Agile Software Development is about the mushy stuff of values and culture.

For example, I think that, ultimately, XP has mushroomed in use and interest not because of pair programming or refactoring but because, taken as a whole, the practices define a developer community freed from the baggage of Dilbertesque corporations. Kent Beck tells the story of an early job in which he estimated a programming effort to be six weeks for two people. After his manager reassigned the other programmer at the beginning of the project (effectively halving the resources), Kent completed the project in twelve weeks--and felt terrible! The boss, of course, harangued Kent throughout the second six weeks. Somewhat despondent because he was such a "failure" as a programmer, Kent finally realized that his original estimate of six weeks was extremely accurate--for two people--and that his "failure" was really his manager's failure to take the responsibility for removing project resources. This type of situation goes on every day with individuals who don't want to make hard tradeoff decisions, so they impose irrational demands.

The Agile movement is not anti-methodology; in fact, we seek to restore credibility to the concept of methodology. We want to restore a balance. We accept modeling, but not in order to file some diagram in a dusty corporate repository. We accept documentation, but not hundreds of pages of never-maintained and rarely used tomes. We plan, but recognize the limits of planning in a turbulent environment. Those who would brand proponents of FDD or Scrum or any of the other Agile approaches as "hackers" are ignorant of both the approaches and the original definition of the term hacker--a programmer who enjoys solving complex programming problems, rather than one who practices either ad hoc development or "cracking." Agile Software Development incorporates proven software engineering techniques, but without the overhead of traditional methodologies.

The Agile Alliance was born in early 2001, but the history of the various approaches and the people who developed them goes back 10 to 15 years. This book describes these practices and the principles behind them, but more important, it delves into the people--the people who are developing the practices and the people who use them to deliver business value to their customers.

Finding a Balance

The left side of the Agile Manifesto value statements indicates what Agilists consider more important than the items on the right side. This should not be construed as indicating that tools, process, documentation, or contracts are unimportant. There is a tremendous difference between one thing being more or less important than another and being unimportant. Judicious use of tools is absolutely critical to speeding software development and reducing its costs. Contracts are one vital component to understanding developer-customer relationships. Documentation, in moderation, aids communication, enhances knowledge transfer, preserves historical information, and fulfills governmental and legal requirements.

But Agilists take a certain perspective on these topics. Try this exercise. For each of the Manifesto value statements, construct two questions along the lines of the following:

  • Could we have a successful project by delivering documentation without working software?
  • Could we have a successful project by delivering working software without any documentation?

By looking at these two end points, we can better grasp relative importance. Although there has to be a balance--documentation and working software, contracts and collaboration, responsiveness and planning, people and process--we have to delineate the extremes, the end points, so that organizations, teams, and individuals can find their own balance points. If we start out trying to find the middle ground, we never will.

One of the great contributions of XP's "extremoes"--Kent Beck, Ron Jeffries, and Ward Cunningham--is that they have staked out positions that have stirred debate in ways that taking moderate positions never would. When Ron says, "Great designs emerge from zero anticipatory design and continuous refactoring," he is challenging himself, and us, to rethink our assumptions about software development. We have to understand the limits before we can understand the balance points.

So although I realize the value of documentation, contracts, plans, and processes, there are numerous sources of material about these subjects. My intention is to identify and define Agile Software Development, to articulate its practices and principles, so you can make your own decision about where on the spectrum you, or your organization, need to be.

Fundamental Questions

There are three fundamental questions that this book answers: (1) What kinds of problems does agility solve best? (2) What is agility? and (3) What are Agile Software Development Ecosystems (ASDEs)?

What Kinds of Problems Does Agility Solve Best?

Problems characterized by change, speed, and turbulence are best solved by agility. Some people argue that good practices are good practices (pair programming, customer focus groups, or feature planning, for example) and therefore ASDEs should not be "limited" to a particular problem type. While true in part, the question asks what problems Agile practices best solve, not just solve. So, while XP, Crystal, or Scrum can surely be used for a wide range of projects, they are particularly relevant to extreme or complex projects--those that have an accelerated time schedule combined with significant risk and uncertainty that generate constant change during the project.

As the level of change caused by rapidly evolving technology, business models, and products increases and the need for delivery speed accelerates, ASDEs' effectiveness increases quickly over rigorous methodologies.

What Is Agility?

Agility, like any other complex concept, has a number of definitions, but for me, the most clearly focused definition is

Agility is the ability to both create and respond to change in order to profit in a turbulent business environment.

Rather than shrink from change, Agile organizations harness or embrace change by being better than competitors at responding to changing conditions and by creating change that competitors can't respond to adequately. However, companies must determine what level of agility they require to remain competitive. Agility is only an advantage relative to competitors--a copper mining company doesn't need to be as agile as a biotechnology firm.

Other aspects of agility are also important: nimbleness or flexibility on the one hand, and balance on the other. Agile organizations are nimble (able to change directions quickly) and flexible (able to see how things that worked for them last week may not work as well next week). An Agile organization also knows how to balance structure and flexibility. If everything changes all the time, forward motion becomes problematic. Agile organizations understand that balancing on the edge between order and chaos determines success.

What Are Agile Software Development Ecosystems?

I began writing this book about Agile Software Development methodologies, but I kept worrying about the word "methodology" because it didn't fit with the focal points of Agile development--people, relationships, and uncertainty. Furthermore, by using the word "methodology," Agile practices are instantly compared to traditional software development methodologies--thereby using the wrong measuring stick for comparison. So I use the term "Agile Software Development Ecosystem" to describe a holistic environment that includes three interwoven components--a "chaordic" perspective, collaborative values and principles, and a barely sufficient methodology--and the term Agilists to identify those who are proponents of ASDEs.

Some people think that "Agile" means fewer processes, less ceremony, and briefer documents, but it has a much broader perspective, which is the primary reason for using the word ecosystem rather than methodology. Although fewer processes and less formality might lower development costs, they are not enough to produce agility. Focusing on people and their interactions and giving individuals the power to make quick decisions and to self-adapt their own processes are key to Agile ecosystems.

The American Heritage Dictionary defines ecosystem as "organisms and their environment: a localized group of interdependent organisms together with the environment that they inhabit and depend on." The Oxford English Dictionary extends this definition to include a constant interchange within the system, including both organic and inorganic elements. The word methodology conjures up a vision of things--activities, processes, tools. These are not inconsequential elements, but incomplete ones. The word ecosystem conjures up a vision of living things and their interactions with each other. Within an organizational context, an ecosystem can then be thought of as a dynamic, ever-changing environment in which people and organizations constantly initiate actions and respond to each other's actions. The word ecosystem focuses us on the dynamic interactions of individuals and teams rather than on the static lines on organization charts.

A Chaordic Perspective

To fully understand ASDEs, we need to understand each of the three components and how they relate to each other. First, Agilists share a view that organizations are chaordic--that every organization exhibits properties of both chaos and order that defy management through the use of linear, predictive planning and execution practices. Viewing organizations as chaordic means understanding that the predictability upon which traditional project management and development life cycle practices are built is a root cause of dysfunctionality between customer, management, and development organizations. A chaordic perspective impacts both how we respond to change and how we manage project teams and organizations.

In day-to-day project work, a chaordic perspective creates two outcomes that are 180 degrees out of sync with rigorous methodologies.

  • Product goals are achievable, but they are not predictable.
  • Processes can aid consistency, but they are not repeatable.

Although ASDEs involve careful planning, the fundamental assumption remains that plans, in a turbulent environment, are not predictable, at least at the level of project scope, schedule, and cost. Plans are hypotheses to be tested rather than predictions to be realized. However, the product goals of the business are achievable, in large part because Agile people adapt. They can "adapt" to an articulated vision and a schedule, scope, or cost goal through tradeoffs in the other two dimensions. Second, while process can aid people in working together, in volatile environments the idea of driving out process variation through measurement and correction--statistical process control--becomes an unworkable hypothesis. Changes that are the result of knowledge gained during the project, knowledge not discernable early in the project, require processes that can respond to change, not ones that attempt to eliminate it.

As Martin Fowler points out, the two fundamental characteristics of ASDEs are their focus on adaptability rather than predictability and on people rather than process (Fowler 2000). As you read through this book, you will see just how fundamental--and challenging to the status quo--these two principles really are. Being Agile means accepting that outcomes are not predictable and that processes are not repeatable. It even implies that as process repeatability increases, project predictability decreases.

Peter Senge uses the term "mental model" to identify the perspective, set of assumptions, stories, and beliefs that each of us carries in our mind that provide a context for thinking (Senge 1990). In organizations, the collective set of mental models defines an overall cultural context. Companies that are heavily sales oriented differ from those that are heavily engineering oriented. Companies whose driving strategy is customer intimacy differ from those whose driving force is product innovation. Companies whose mental model includes linearity, cause and effect, hierarchy, predictability, and control will operate very differently from one whose mental model includes collaborative networks, emergence, decentralization of power, and acceptance of unpredictability. One is Newtonian, the other chaordic.

A chaordic perspective draws on a complex adaptive systems model in which decentralized, independent agents (each of us) interact in self-organizing ways, guided by a set of simple, generative rules that create complex, emergent results. This perspective is examined in detail in my book Adaptive Software Development (Highsmith 2000) and is explicitly embraced in the philosophies of Agilists Ken Schwaber, Bob Charette, and Kent Beck.

XP provides an example of how methodology and culture fit together. At one level, XP is defined by a system of practices--pair programming, test-first development, refactoring, coding standards. But the values and principles of XP define a collaborative culture--how developers work together with customers, how individuals interact and treat each other as human beings.

Collaborative Values and Principles

The second piece of the interconnected web that defines ASDEs is the statement of collaborative values and principles. While it is difficult to characterize the Agile Manifesto in one word, "collaborative" seems to be the best single adjective. Values and principles shape the ecosystem. Without a set of stated values and principles, an ecosystem is sterile, reflecting practices but not the people who interact within it.

A collaborative culture includes people and their relationships within a development team and with customers, management, and partnering teams within or external to their own company. Human dynamics, communications, and collaboration may be known as the "soft" sciences, but in practice, they may be the hardest to master. Principles and values help define a culture--the environment in which people want to work.

A Barely Sufficient Methodology

The final component of an ASDE is methodology. The traditional definition of methodology includes things such as roles, activities, processes, techniques, and tools. Alistair Cockburn summarizes these components when he defines methodology as "the conventions we agree to"--the ways in which people work together on a project. In The Social Life of Information, John Seely Brown and Paul Duguid (2000) discuss the major differences between process (as used by the business process reengineering movement) and practice. Processes are described in manuals; practices are what happen in reality. Process centrists relegate people to second place; practice centrists place people first. Process centrists focus on explicit (written down) knowledge, while practice centrists focus on tacit (internal) knowledge. The ASDE model provides a practice-centered, rather than a process-centered, approach to methodology.

There are two reasons to pursue barely sufficient methodologies: value and innovation. Streamlined methodologies concentrate on those activities that create value and ruthlessly eliminate non-value-adding activities. Programming usually adds value; process management often adds overhead. Bare sufficiency means keeping the former and eliminating the latter. Second, innovation and creativity flourish in chaordic environments, not orderly ones. Barely sufficient methodologies are cauldrons for breeding innovation.

Methodology also relates to organizational model. Agile methodologies contain minimal processes and documentation and reduced ceremony (formality). Agile methodologies are labeled "barely sufficient" (Cockburn) or "a little bit less than just enough" (Highsmith), or "minimal" (Charette). However, this streamlining of methodology isn't based just on reducing work effort but, more important, it is based on understanding the chaordic world view--one in which emergent (innovative) results are best generated at the "edge of chaos," perched midway between chaos and order.

Practices (or techniques) are the lifeblood of methodology. Whether it's pair programming, Scrum meetings, customer focus groups, or automated testing, the practices of ASDEs, carried out by talented and skilled individuals, produce results.

Changing Perspectives

In the software profession, we've used the words "methodology" and "process" for so long that they roll off the tongue and the pen without trouble. "Ecosystem" will take getting used to, but then, that's why I'm using the word--to foster a different perspective. "There is accumulating evidence that corporations fail because the prevailing thinking and language of management are too narrowly based on the prevailing thinking and language of economics," says Arie De Geus (1997). "They forget that their organizations' true nature is that of a community of humans."

To change thinking, we must change the language we use, so I use the word "ecosystem" to change our thinking about how software projects should be viewed. A chaordic perspective, a collaborative set of values and principles, and a barely sufficient methodology all combine and interact to form an Agile ecosystem. We cannot separate the three, at least in my mind, and I think most Agilists would agree. One could have a streamlined methodology but a linear, Newtonian view of organizations, and the result would not be Agile. One could have a streamlined methodology but a hierarchical, control-based work culture, and it would not be Agile. One could have a collaborative, people-oriented work culture but a rigid, predictive approach to planning and managing projects, and it would not be Agile. Agility requires all three.

The ultimate objective of this book is to describe new ways of working together to deliver business value to software customers. The heart of ASDEs is a core belief in people--their individuality and their interactions. It's impossible to discuss people and their ways of working together (ecosystem) without discussing values and principles. It's impossible to discuss values and principles without also discussing assumptions about how organizations do, or should, work. It's impossible to compare Agile approaches with non-Agile approaches using "methodology" as the only mechanism for comparison.

In closing, I need to state that I don't speak for anyone in the Agile community other than myself. I may interpret what Ken Schwaber says about Scrum, what Jeff De Luca says about FDD, or what Alistair Cockburn says about Crystal Methods, but they are my interpretations. In making generalizations about ASDEs, I'm sure I've made statements that would generate disagreement from 1 or more of the other 16 authors of the Agile Manifesto. However, I have talked to, corresponded with, or worked with all these authors, so although they are my own interpretations, they also reflect a deep sense of being part of, and contributor to, this community.

Although 17 individuals authored the Agile Manifesto, thousands support this effort. Many, many individuals have signed the Manifesto Web page, and an array of Web sites prominently display the statement "We support the Agile Manifesto." Agilists have stirred a healthy debate within the software development and project management communities. I hope this book will contribute to that debate and encourage others to join in it.

Jim Highsmith
Salt Lake City, Utah

0201760436P04092002

Read More Show Less

Introduction

From February 11 to 13, 2001, at The Lodge at Snowbird ski resort in the Wasatch mountains of Utah, seventeen people met to talk, ski, relax, and try to find common ground. What emerged was the Agile Software Development movement. Representatives from Extreme Programming (XP), Scrum, the Dynamic Systems Development Method (DSDM), Adaptive Software Development (ASD), Crystal Methods, Feature-Driven Development (FDD), Pragmatic Programming, and others sympathetic to the need for an alternative to document-driven, rigorous software development processes convened. What emerged from this meeting was symbolic—a Manifesto for Agile Software Development—signed by all seventeen of the participants:

We are uncovering better ways of developing software by doing it and helping others do it. We value:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan.

That is, while there is value in the items on the right, we value the items on the left more.

This value statement has a number of fascinating aspects, not the least of which was getting seventeen people to agree to it. Although this was a group of experienced and recognized software development "gurus," the word uncovering was selected to indicate that the authors don't have all the answers and don't subscribe to the silver-bullet theory.

The phrase "by doing it" indicates that the authors actually practice these methods in their own work. Ken Schwaber told thstory of his days of selling tools to automate comprehensive, "heavy" methodologies. Impressed by the responsiveness of Ken's company, Jeff Sutherland asked him which of these rigorous methodologies he used for internal development. "I still remember the look on Jeff's face," Ken remarked, "when I told him—none—if we used any of them, we'd be out of business." The authors want to help others with Agile practices and to further our own knowledge by learning from those we try to help.

The value statements have a form: In each bullet point the first segment indicates a preference while the latter segment describes an item that while important, is of lesser priority. This distinction lies at the heart of agility, but simply asking people to list what's valuable doesn't flesh out essential differences. Roy Singham, CEO of ThoughtWorks, put it well when he said that it's the edge cases, the hard choices, that interest him. "Yes, we value planning, comprehensive documentation, processes and tools. That's easy to say. The hard thing is to ask 'what do you value more?'" When push comes to shove—and it usually does—something must give, and we need to be clear about what stays and what gives.

The first value statement recognizes the importance of processes and tools, and stresses that the interaction of talented individuals is of far more importance. Rigorous methodologies and their business process-engineering brethren place more emphasis on process than people. Agile development reverses this trend. If individuals are unique, and we believe they are, then processes should be melded to individuals and teams, not the other way around.

Similarly, while comprehensive documentation is not necessarily harmful, the primary focus must remain on the final product—working software. This means that every project team needs to determine, for itself, what documentation is absolutely essential to delivering working software. Working software tells the developers and sponsors what they really have in front of them—as opposed to promises as to what they might have in front of them. Working software can be shipped, modified, or scrapped, but it is always real.

Contract negotiation, whether through an internal project charter or an external legal contract, isn't a bad practice, just a seriously insufficient one. Customers want a product that conforms to the his or her needs—at the time of delivery. "Contract negotiation encourages the addition of buffers (contingency)," says colleague Ron Holliday. "This makes projects take even longer, drives up costs, and reduces responsiveness to change. A collaborative arrangement is a team effort between customer and supplier to deliver the best possible solution." Contracts and project charters provide boundary conditions within which the parties can work, but only through ongoing collaboration can a development team hope to understand and deliver what the client wants.

No one can argue that following a plan is a good idea—right? Well, yes and no. In a world of business and technology turbulence, scrupulously following a plan can have dire consequences, even if it's executed faithfully. However carefully a plan is crafted, it becomes dangerous if it blinds you to change. As you will see in several of the case studies presented in this book, few, if any, of the projects delivered what was planned for in the beginning. And yet, they were successful because the development team was Agile enough to respond again and again to external changes. In the Information Age, planning is important, but accepting that deviations from any plan as are "normal" is even more important.The meeting at Snowbird was incubated at a spring 2000 gathering of Extreme Programming leaders organized by Kent Beck. At that meeting in Oregon, a few "outsiders" but "sympathizers," like myself, were invited and attendees voiced support for a variety of "light" methodologies. During 2000 a number of articles were written that referenced the category of "light" or "lightweight" practices. In conversation, the participants didn't like the moniker "light," but it stuck at that time.

In September 2000, "Uncle Bob" Martin of Object Mentor in Chicago, started what was to become the Agile ball rolling with an email; "I'd like to convene a small (two day) conference in the January to February 2001 timeframe here in Chicago. The purpose of this conference is to get all the lightweight method leaders in one room. All of you are invited; and I'd be interested to know who else I should approach." Bob set up a Wiki site and the discussions raged. Early on, Alistair Cockburn weighed in with an epistle that identified the general disgruntlement with the word 'light': "I don't mind the methodology being called light in weight, but I'm not sure I want to be referred to as a lightweight attending a lightweight methodologists meeting. It somehow sounds like a bunch of skinny, feeble-minded lightweight people trying to remember what day it is."

The fiercest debate was over location! There was serious concern about Chicago in wintertime—cold and nothing fun to do. Someone suggested Snowbird, Utah—cold, but fun things to do, at least for those who ski on their heads like Martin Fowler tried on the first day. Someone else mentioned Anguilla in the Caribbean—warm and fun, but time consuming to get to. In the end, Snowbird and skiing won out.

The Agile Manifesto value statements represent a deeper theme that drives many, but not all, of its authors. At the close of the two-day meeting, Bob Martin joked that he was about to make a "mushy" statement. Although tinged with humor, no one disagreed with Bob's sentiments—that we all felt privileged to work with a group of people who held a set of compatible values, based on trust and respect for each other, promotion of organizational models based on people, and the desire to build collaborative communities in which to work. At the core, I believe Agile developers are really about "mushy" stuff—about delivering products to customers while operating in a vibrant, sustaining work place. So in the final analysis, the meteoric rise of interest in—and criticism of—Agile Software Development is about the mushy stuff of values and culture.

For example, I think that, ultimately, XP has mushroomed in use and interest, not because of pair programming or refactoring, but because taken as a whole, the practices define a developer community freed from the baggage of Dilbertesque corporations. Kent Beck tells the story of an early job in which he estimated a programming effort of six weeks for two people. After his manager reassigned the other programmer at the beginning of the project (effectively halving the estimate), Kent completed the project in twelve weeks—and felt terrible! The boss, of course, harangued Kent throughout the second six weeks. Kent, somewhat despondent because he was such a "failure" as a programmer, finally realized that his original estimate of six weeks was extremely accurate—for two people—and that his "failure" was really his manager's failure. This type of situation goes on every day with individuals who don't want to make hard trade-off decisions, so they impose irrational demands.

The Agile movement is not anti-methodology; in fact, we seek to restore credibility to the concept of methodology. We want to restore a balance. We embrace modeling, but not to file some diagram in a dusty corporate repository. We embrace documentation, but not hundreds of pages of never-maintained and rarely used tomes. We plan, but recognize the limits of planning in a turbulent environment. Those who would brand proponents of FDD or Scrum or any of the other Agile approaches as "hackers" are ignorant of both the approaches and the original definition of the term hacker— programmers who enjoy solving complex programming problems rather than those who practice ad hoc development or "cracking." Agile Software Development incorporates proven software engineering techniques, but without the overhead of traditional methodologies.

The Agile Alliance was born in early 2001, but the history of the various approaches and the people who developed them goes back ten to fifteen years. This book describes these practices and the principles behind them, but more important, it delves into the people—the people who are developing the practices, and the people who use them to deliver business value to their customers.

Finding a Balance

The left side of the Agile Manifesto value statements indicates what Agilists consider more important than the item on the right side. This should not be construed as indicating that tools, process, documentation, or contracts are unimportant. There is a tremendous difference between one thing being more or less important than another and being unimportant. Judicious use of tools are absolutely critical to speeding software development and reducing its costs. Contracts are one vital component to understanding developer-customer relationships. Documentation, in moderation, aids communication, enhances knowledge transfer, preserves historical information, and fulfills governmental and legal requirements.

But Agilists take a certain perspective on these topics. Try this exercise. For each of the Manifesto value statements, construct two questions along the lines of the following:

  • Could we have a successful project by delivering a documentation without working code?
  • Could we have a successful project by delivering working code without any documentation?

By looking at these two end points, we can better grasp relative importance. While there has to be a balance—documentation and working software, contracts and collaboration, responsiveness and planning, people and process—we have to delineate the extremes, the end points, in order that organizations, teams, and individuals can find their own balance points. If we start out trying to find the middle ground—we never will.

One of the great contributions of XP's "extremoes"—Kent, Ron, and Ward—is that they have staked out positions th have stirred debate in ways that taking moderate positions never would. When Ron says, "Great designs emerge from zero anticipatory design and continuous refactoring," he is challenging himself, and us, in ways that taking a moderate position never would. We have to understand the limits before we can understand the balance points.

So although I realize the value of documentation, contracts, plans, and processes, there are numerous sources for material about these subjects. My intention is to identify and define Agile Software Development, articulate it's practices and principles, so you can make your own decision about where on the spectrum you, or your organization, need to be.

Fundamental Questions

There are three fundamental questions that this book answers: (1) What kinds of problems does agility solve best? (2) What is agility? And, (3) What are Agile Software Development Ecosystems (ASDEs)?

What Kinds of Problems Does Agility Solve Best?

Problems characterized by change, speed, and turbulence are best solved by agility. Some people argue that good practices are good practices (pair programming, customer focus groups, or feature planning, for example) and therefore ASDEs should not be "limited" to a particular problem type. While true in part, the question asks about what problems Agile practices best solve, not just solve. So, while XP, Crystal, or Scrum can surely be used for a wide range of projects, they are particularly relevant to extreme or complex projects—those that have an accelerated time schedule combined with significant risk and uncertainty that generate constant change during the project.

As the level of change caused by rapidly evolving technology, business models, and products increases, ASDEs' effectiveness increases quickly over traditional rigorous methodologies.

What Is Agility?

Agility, like any other complex concept, has a number of definitions, but for me, the most clearly focused definition is:

Agility is the ability to both create and respond to change in order to profit in a turbulent business environment.

Rather than shrink from change, Agile organizations harness or embrace change by being better than competitors at responding to changing conditions and by creating change that competitors can't respond to adequately. However, companies must determine what level of agility they require to remain competitive. Agility is only an advantage relative to competitors—a copper mining company doesn't need to be as agile as a biotechnology firm.

Two other aspects of agility are also important: nimbleness or flexibility, and balance. Agile organizations are nimble, able to change directions quickly, and flexible, able to see how things that worked for them last week, may not work as well next week. An Agile organization also knows how to balance structure and flexibility. If everything changes all the time, forward motion becomes problematic. Agile organizations understand that balancing on the edge between order and chaos determine success.

What Are Agile Software Development Ecosystems?

I began writing this book about Agile Software Development methodologies, but I kept worrying about the word "methodology" because it didn't fit with the focal points of Agile development—people, relationships, and uncertainty.

Furthermore, by using the word "methodology," Agile practices are instantly compared to traditional software development methodologies—thereby using the wrong measuring-stick for comparison. So I use the term "Agile Software Development Ecosystems" to describe a holistic environment that includes three interwoven components—a chaordic perspective, collaborative values and principles, and a barely sufficient methodology—and the term Agilists to identify those who are proponents of ASDEs.

Some people think that Agile means fewer processes, less ceremony, and briefer documents, but it has a much broader perspective, which is the primary reason for using the word ecosystem rather than methodology. Although fewer processes and less formality might lower development costs, they are not enough to produce agility. Focusing on people and their interactions, giving individuals the power to make quick decisions and to self-adapt their own processes are key to Agile ecosystems.

American Heritage Dictionary defines ecosystem as "organisms and their environment: a localized group of interdependent organisms together with the environment that they inhabit and depend on." The Oxford English Dictionary extends this definition to include a constant interchange within the system, including both organic and inorganic elements. The word methodology conjures up a vision of things—activities, processes, tools—not inconsequential elements, but incomplete ones. Methodology deals with things; ecosystems deal with people. The word ecosystem conjures up a vision of people and their interactions with each other. An ecosystem is a dynamic, ever-changing environment in which people and organizations constantly initiate actions and respond to other's actions.

A Chaordic Perspective

To fully understand ASDEs, we need to understand each of three components and how they interact. First, Agilists share a view that organizations are chaordic—that every organization exhibits properties of both chaos and order that defy management through the use of traditional linear, predictive planning and execution practices. Viewing organizations as chaordic means understanding that the predictability upon which traditional project management and development life cycle practices are built is a root cause of dysfunctionality between customer and development organizations. A chaordic perspective impacts both how we respond to change and how we manage project teams and organizations.

To day-to-day project work, a chaordic perspective creates two outcomes that are 180 degrees out of sync with traditional methodologies:

  • Product goals are achievable, but not predictable.
  • Processes can aid consistency, but they are not repeatable.

While ASDEs involve careful planning, the fundamental assumption remains that plans, in a turbulent environment, are not predictable, at least at the traditional level of project scope, schedule, and cost. However, the business' product goals are achievable, in large part because Agile processes adapt. They can "adapt" to a schedule, scope, or cost goal through trade offs in the other two dimensions. Second, while process can aid people in working together, in volatile environments the idea of driving out process variation through measurement and correction—statistical process control—becomes an unworkable hypothesis.

As Martin Fowler points out in his article "New Methodology," the two fundamental characteristics of ASDEs are their focus on adaptability rather than predictability, and, that people are rather than process. As you read through this book you will see just how fundamental—and challenging to the status quo—these two principles really are. Being Agile means accepting that outcomes are not predictable and that processes are not repeatable. It even implies that as process repeatability increases, project predictability decreases.

Peter Senge uses the term "mental model" to identify the perspective, set of assumptions, stories, and beliefs that each of us carries in our minds that provide a context for thinking. In organizations, the collective set of mental models defines an overall cultural context. Companies that are heavily sales oriented differ from those that are heavily engineering oriented. Companies whose driving strategy is customer intimacy differ from those whose driving force is product innovation. Companies whose "mental model" includes linearity, cause-and-effect, hierarchy, predictability, and control, will operate very differently that one whose "mental model" includes collaborative networks, emergence, decentralization of power, and acceptance of unpredictability. One is Newtonian, the other Chaordic.

A chaordic perspective draws on a complex adaptive systems model in which decentralized, independent agents (each of us) interact in self-organizing ways, guided by a set of simple, generative rules, that create complex, emergent results. This perspective is examined in detail in Adaptive Software Development, and is explicitly embraced in the philosophies of Agilists Ken Schwaber, Bob Charette, and Kent Beck.

XP provides an example of how methodology and culture fit together. At one level, the practices of XP are defined by a system-of-practices—pair programming, test-first development, refactoring, coding standards. But the values and principles of XP define a collaborative culture—how developers work together with customers, how individuals interact and treat each other as human beings.

Collaborative Values and Principles

The second piece of the interconnected web that define ASDEs is the statement of collaborative values and principles. While it is difficult to characterize the Agile Manifesto in one word, "collaborative" seems to be the best single adjective. Values and principles shape the ecosystem. Without a set of stated values and principles, an ecosystem is sterile, reflecting practices but not the people who interact within it.

A collaborative culture includes people and their relationships within a development team, their relationships to customers, and their relationships with partnering teams within or external to their own company. Human dynamics, communications, and collaboration may be known as the "soft" sciences, but in practice, they may be the "hardest" to master. Principles and values help define a culture—the environment in which people want to work.

A Barely Sufficient Methodology

The final component of an ASDE is methodology. The traditional definition of methodology includes things such as roles, activities, processes, techniques, and tools. Alistair summarizes these components when he defines methodology as "the conventions we agree to,"—the ways in which people work together on a project. In The Social Life of Information, John Seely Brown and Paul Duguid discuss the major differences in process (as used by the business-process reengineering movement), and practice. Processes are described in manuals; practices are what happen in reality. Process-centrist relegate people to second place; practice-centrists place people first. Process-centrists focus on explicit knowledge (written down), while practice-centrists focus on tacit (internal) knowledge. The ASDE model provides a practice-centered, rather than a process-centered, approach to methodology.

There are two reasons to pursue barely sufficient methodologies—value and innovation. Streamlined methodologies concentrate on those activities that create value and ruthlessly eliminate non-value adding activities. Programming usually adds value, process management often adds overhead. Bare sufficiency means keeping the former and eliminating the latter. Second, innovation and creativity flourish in chaordic environments, not orderly ones. Barely sufficient methodologies are cauldrons for breeding innovation.

Methodology also relates to organizational model. Agile methodologies contain minimal processes and documentation and reduced ceremony (formality). Agile methodologies are labeled "barely sufficient" (Cockburn) or "a little bit less than just enough" (Highsmith), or "minimal" (Charette). However, this streamlining of methodology isn't just based on reducing work effort, but more importantly, it is based on understanding the chaordic world view—one in which emergent (innovative) results are best generated at the "edge of chaos," perched midway between chaos and order.

Practices (or techniques or methods) are the lifeblood of methodology. Whether its pair programming, Scrum meetings, customer focus groups, or automated testing, the practices of ASDEs, or rigorous methodologies for that matter, wielded in the hands of talented and skilled individuals, produce results.

Changing Perspectives

In the software profession, we've used the words "methodology" and "process" for so long that they roll off the tongue and the pen without trouble. "Ecosystem" will take getting used to, but then, that's why I'm using the word—to foster a different perspective. "There is accumulating evidence that corporations fail because the prevailing thinking and language of management are too narrowly based on the prevailing thinking and language of economics," says Arie De Geus. "They forget that their organizations' true nature is that of a community of humans."

To change thinking, we must change the language we use, so I use the word and "ecosystem" to change our thinking about how software projects should be viewed. A chaordic perspective, a collaborative set of values and principles, and a barely sufficient methodology all combine and interact to form an Agile ecosystem. We cannot separate the three, at least in my—and I think in most Agilists—minds. One could have a streamlined methodology, but a linear, Newtonian view of organizations, and the result would not be Agile. One could have a streamlined methodology, but a hierarchical, control-based work culture, and it would not be Agile. One could have a collaborative, people-oriented work culture, but a rigid, predictive approach to planning and managing projects, and it would not be Agile. Agility requires all three.

The ultimate objective of this book is to describe new ways of working together to deliver business value to software customers. The heart of ASDEs involve a core belief in people—their individuality and their interactions. It's impossible to discuss people and their ways of working together (ecosystem) without discussing values and principles. It's impossible to discuss values and principles without also discussing assumptions about how organizations do, or should, work. It's impossible to compare Agile approaches with non-Agile approaches using "methodology" as the only mechanism for comparison.

In closing, this preface I need to state that I don't speak for anyone in the Agile community other than myself. I may interpret what Ken Schwaber says about Scrum, what Jeff De Luca says about FDD, or what Alistair Cockburn says about Crystal Methods, but they are my interpretations. In making generalizations about ASDEs, I'm sure I've made statements that would generate disagreement from one, or more of the other sixteen authors of the Agile Manifesto. However, I have talked to, corresponded with, or worked with all the authors of these ecosystems, so although they are my own interpretations, they also reflect a deep sense of being a part of, and contributor to, this community.

Although seventeen individuals authored the Agile Manifesto,thousands support this effort. Many, many individuals have signed the Manifesto Web page, and an array of Web sites prominently display "We support the Agile Manifesto." Agilists have stirred a healthy debate within the software development and project management communities. Hopefully this book will contribute to that debate and encourage others to join in it.

Jim Highsmith
Salt Lake City
January 2002


Read More Show Less

Customer Reviews

Be the first to write a review
( 0 )
Rating Distribution

5 Star

(0)

4 Star

(0)

3 Star

(0)

2 Star

(0)

1 Star

(0)

Your Rating:

Your Name: Create a Pen Name or

Barnes & Noble.com Review Rules

Our reader reviews allow you to share your comments on titles you liked, or didn't, with others. By submitting an online review, you are representing to Barnes & Noble.com that all information contained in your review is original and accurate in all respects, and that the submission of such content by you and the posting of such content by Barnes & Noble.com does not and will not violate the rights of any third party. Please follow the rules below to help ensure that your review can be posted.

Reviews by Our Customers Under the Age of 13

We highly value and respect everyone's opinion concerning the titles we offer. However, we cannot allow persons under the age of 13 to have accounts at BN.com or to post customer reviews. Please see our Terms of Use for more details.

What to exclude from your review:

Please do not write about reviews, commentary, or information posted on the product page. If you see any errors in the information on the product page, please send us an email.

Reviews should not contain any of the following:

  • - HTML tags, profanity, obscenities, vulgarities, or comments that defame anyone
  • - Time-sensitive information such as tour dates, signings, lectures, etc.
  • - Single-word reviews. Other people will read your review to discover why you liked or didn't like the title. Be descriptive.
  • - Comments focusing on the author or that may ruin the ending for others
  • - Phone numbers, addresses, URLs
  • - Pricing and availability information or alternative ordering information
  • - Advertisements or commercial solicitation

Reminder:

  • - By submitting a review, you grant to Barnes & Noble.com and its sublicensees the royalty-free, perpetual, irrevocable right and license to use the review in accordance with the Barnes & Noble.com Terms of Use.
  • - Barnes & Noble.com reserves the right not to post any review -- particularly those that do not follow the terms and conditions of these Rules. Barnes & Noble.com also reserves the right to remove any review at any time without notice.
  • - See Terms of Use for other conditions and disclaimers.
Search for Products You'd Like to Recommend

Recommend other products that relate to your review. Just search for them below and share!

Create a Pen Name

Your Pen Name is your unique identity on BN.com. It will appear on the reviews you write and other website activities. Your Pen Name cannot be edited, changed or deleted once submitted.

 
Your Pen Name can be any combination of alphanumeric characters (plus - and _), and must be at least two characters long.

Continue Anonymously

    If you find inappropriate content, please report it to Barnes & Noble
    Why is this product inappropriate?
    Comments (optional)