Agile methodologies have become a popular and widely accepted method for managing software development. However, despite this success, managing agile methods has proven to be a real challenge for most companies, particularly those with complex products such as IoT devices and large development environments. Many companies have been forced to adopt a hybrid version of agile and waterfall techniques, and this hybrid approach is fast becoming the norm rather than the exception in the industry. Metagility is the first book to provide a comprehensive approach for managing a new and highly effective breed of agility from the executive level on down. Based on scientific theory and practitioner research, it is the definitive playbook for those seeking the optimal solution for adapting agile to more complex product development and organizational contexts.
|Publisher:||Ross, J. Publishing, Incorporated|
|Product dimensions:||6.00(w) x 9.00(h) x 0.80(d)|
About the Author
Dr. David A. Bishop is a technology consultant, enterprise architect, researcher, and instructor with over 25 years of experience.
Read an Excerpt
AGILE METHODS TODAY: WHAT HAS CHANGED AND WHY
Agile methodologies have become a popular and widely respected method for managing software development. Since the inception of the Agile Manifesto, agile development methodologies have superseded waterfall methodology in many, if not most, software development organizations. Despite its apparent success, however, many organizations have struggled with the adoption and implementation of agile and exactly what level of adoption provides the most agility. Agility is commonly held in the literature to be constructed of elements external to a company or project but, in fact, may be composed of both external and internal elements. The exact relationship of the adoption of agile development techniques to a company or organization's true agility remains unclear. Crucially, there are no measurements available for each of these factors in an embedded software development context nor is there a method to determine the impact of one against the other. A major reason for the lack of measurements is due to the somewhat amorphous definition of agile itself. In academic literature, the concept is still relatively young and poorly defined. In practice, organizations have largely opted for a hybrid approach to agile, mixing its concepts and method with existing stage-gate or waterfall methodologies. This has made the management of agile even more complex. These issues beg the following questions:
How do organizations orchestrate and assess agility?
How is agile best managed in difficult or complex situations?
How can some of the commonly known shortcomings of agile be mitigated?
Most important, given the apparent efficiency and performance gains of agile, how can it be leveraged to maximize organizational performance and product success?
Metagility answers these questions by examining multiple dimensions of the issue (agile definition, history, adoption, implementations, hybrid solutions, project/program management) to determine how agility processes are orchestrated (meta-agility) in a number of companies that were successful in leveraging and maximizing agility to become number one in their respective market(s). Throughout this book, we highlight one of these case studies in particular. A study of an embedded systems development organization was conducted with twelve managers from hardware, firmware, and software development groups within the company. These managers included product, project, people, and technology managers with direct responsibility for managing agility processes. An innovative form of research, referred to as grounded theory, was utilized to perform data analysis and determine a central phenomenon or theory at the crux of the agile question. This research revealed the central phenomenon to be agile vorticity — the point at which maximum agility is achieved. This point is reached through the management (meta-agility) of two main subcategories: process and market agility. The relationships of these categories are illustrated with a thought experiment of a whirlpool vortex. Included is an empirical account of how agility can be managed and tweaked for optimum results. In the chapters that follow, we examine market and process agility to determine the success factors for achieving agile vorticity and becoming number one in your market!
We begin by establishing a historical perspective of agility — its evolution and its current state as compared to when it was first developed.
THE AGILE MANIFESTO TODAY
The Agile Manifesto as it was originally written in 2001 is a set of four now very well-known tenets:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
The idea being that while all elements in the previous statements have value, the ones on the left side are valued even more. In this section, we will explore each of these four tenets and how today's environment is influencing them to change.
Individuals and Interactions over Processes and Tools
In 2001, people worked more independently as waterfall processes tended to limit interactions and focus on well-defined goals. Agilists hoped to break this lack of interaction by emphasizing teamwork, but much of this was predicated on colocated teams. As Table 1.1 illustrates, teams today and the communication channels they use are more complex. Many development organizations are distributed all over the world and use a variety of communication methods and tools. With that in mind, tools themselves have become much more sophisticated and user friendly. Many such tools are easily customizable and have agile concepts built in.
Waterfall was extremely heavy with a plethora of burdensome steps and operating procedures designed for CYA (cover your ass!) rather than focusing on creating a great product. On the other hand, some aspects of agile, such as team collaboration, have proven challenging to implement at scale. Although interactions are certainly very important, the lack of process around documenting the outcomes and decisions of these interactions can result in a great deal of churn. Churn in this context means burning resources and losing time due to repetitive meetings and rework.
Many organizations have found that today's tools can facilitate conversations and can be used to drive, not inhibit, development. Additionally, they have also discovered that implementing agile itself is a process, even if it is much smaller in scope than waterfall was. The right combination of process and tools can be decisive in the success of your business.
Working Software over Comprehensive Documentation
Older waterfall processes often resulted in cumbersome and exhaustive word documents which included requirements documentation, design documents, standard operating procedures, maintenance documents, support documents, and much more. Emphasis was placed on writing many of these documents up front before attempting to build anything. This comprehensive documentation tended to inhibit response to change and resulted in a slowdown of innovation and productivity. Documentation is still needed for today's development, but it takes a different form. It is no longer a series of heavy word documents, rather it comes in the form of features, epics, user stories, and other artifacts that may be documented within collaboration tools or other applications. Table 1.2 illustrates these differences. As mentioned previously, outcomes of key interactions need to be documented, just not in a process heavy or cumbersome document, but within a lean and easily managed set of tools.
The need for documentation is even more critical with today's complex products such as smart devices or Internet of Things (IoT) applications that are created using multiple teams, development tracks, and sub-products.
Customer Collaboration over Contract Negotiation
Contracts are a fact of business life. They aren't going away. Regulatory requirements, idiosyncrasies of specific industries, or the nature of certain products require them. In today's world, contracts and collaboration are not mutually exclusive; in fact, they are anything but. When we discuss contract negotiation, that negotiation within itself is a form of collaboration with the customer. The difference is that this collaboration is not necessarily a one-time thing where all items are discussed and documented up front in a contract, rather it is an ongoing process of communication and alignment.
Collaboration has become more than just talking directly to the customer. Often, the customer may not be able to articulate exactly what they need, and it may require a variety of stakeholders on both the customer side and internal to the development organization to glean succinct requirements and break those down into increments that can be developed into a working product. Table 1.3 illustrates this change.
Responding to Change over Following a Plan
In 2001, plans were synonymous with rigid. A plan was often considered to be a representation of the final work. Making changes midstream once the plan was finalized was discouraged, and if changes were implemented, it often resulted in a painful approval process and reevaluation of delivery timelines, costs, and performance expectations. A great deal of work was performed up front to ensure that the plan, and the requirements it entailed, were as complete as possible so that developers knew exactly what they needed to do down to the finest detail. Different interpretations of the plan, unanticipated engineering challenges, and inability to accommodate changing customer needs often resulted in failure. Today, markets are simply moving too fast to adhere to this kind of rigid approach. At the same time, there must be some kind of plan to ensure that the company is getting the business value out of the development work that not only makes the customer happy but also satisfies their strategic goals. This means that the definition of a plan has changed. As Table 1.4 illustrates, no longer is it a long, exhaustive document that is full of requirements and design specifications, rather it is more of a vision for the organization. Change is inevitable, but every change must be evaluated against the values and needs of the business.
Now that we understand how the Agile Manifesto has changed from a practitioner standpoint, let's take a look at what the business research says about the topic.
AGILE MANAGEMENT TODAY: A RESEARCH PERSPECTIVE
The management of agile methodologies can be a challenge to organizations on many levels. The first issue is the concept of agile itself. The current research literature on agility is sparse, particularly in relation to information systems development (ISD). This is largely due to the fact that agile is a relatively new concept in an ISD context and is therefore not entirely solidified. It is, in essence, a set of concepts or ideas, not necessarily a methodology or framework of its own. The second challenge is the management of agility. Most agile-based frameworks available today are operational in nature, focusing on project management indicators such as velocity, release frequency, sprint completion, and so forth (Highsmith, 2010). Still, other methods focus entirely on the outcome of agile adoption in relation to environmental turbulence (Yauch, 2011). This type of study, however, does not allow for the evaluation of the adoption of agile principles, such as people over processes and tools, against quality and customer responsiveness. Giachetti holds that the assessment and management of agile should not only consider performance, but agile characteristics that have been assimilated into the organization (Giachetti et al., 2003). Although there have been methods describing agile adoption, none of these has related the level of adoption to agility outcomes. As a result, these existing methods do not completely address all dimensions of agility. These challenges are compounded by the fact that recent research has revealed that most organizations have not adopted agile in its entirety, but instead have assimilated a variety of agile concepts and methods into existing traditional methods. Such hybrid agile implementations have only added to the complexity of agility management.
Orchestration of agility within any organization is often poorly defined, due to its fluid, quickly changing, and somewhat amorphous concepts, processes, and methodologies. Hybrid implementations, environment specific assessments, and varying degrees of market turbulence are just a few dimensions of agility that should be considered within a management context. In short, the management of agile methods requires their own brand of agility. Our goal from a research perspective was to examine our case studies against the various dimensions of this agility to determine how agile processes are typically managed and orchestrated.
WHY ARE AGILE METHODS AND AGILITY CONCEPTS SO CRITICAL TO BUSINESS SUCCESS NOW?
The short answer to that question is that it is becoming increasingly difficult to stay ahead. Technology today is changing at an ever-accelerating rate. In his essay "The Law of Accelerating Returns," futurist Ray Kurzweil said that this rate of acceleration is occurring exponentially and could eventually result in "technological change so rapid and profound it represents a rupture in the fabric of human history." We see evidence of this change in our everyday lives, not just in technology. Companies must work harder and harder to keep up, tweaking ever more agility out of their processes and teams.
Many case studies in our research experienced hyper-accelerated markets, which created some very unique conditions. One of these conditions is what is referred to as an agile vortex or whirlpool. Agile vortices are created when extreme levels of market pressure are torqued by high innovation or quickly evolving technology. This provided an excellent laboratory under which it was possible to observe how a company achieved optimum performance, or essentially agile nirvana in today's most challenging markets. This phenomenon is referred to as reaching the point of agile vorticity, which will be explained in a later chapter.
How these hyper-accelerated markets take shape can be illustrated by close examination of one of our cases studies. In this case, the technology developers were in the smart grid market, developing smart meters for power utilities that wanted to upgrade their grids to the latest green technologies. These are the same kind of meters that you may find on the side of your house or business; the difference is they are automated, more accurate, and in most cases communicate wirelessly without the need to send out technicians to read them. Not only do smart meters save money and protect the environment, they provide greater security as well. The subject of our case study faced several challenges which could be summarized in the following ways.
1. Carrier and networking technology was rapidly changing: From low power radio frequency (RF) signals to cellular to power line carrier, there are a host of communication methods that could be used to transport meter data from the field to the utility. Adopting the best technology could make or break a company. Our case study had invested deeply into RF mesh networking technology and was constantly faced with new threats.
2. Meter capabilities dramatically increasing: Originally fairly dumb devices, electric, gas, and water meters were acquiring the ability to manage smart devices throughout the home, communicate pricing to customers, and gather detailed event and power data. In many ways, the growing capabilities of the available technologies were outpacing the company's ability to get them tested and implemented — not to mention the fact that competitors were in the same race. The same thing was happening, and in fact still is happening, with other IoT solutions. Our case study was rushing to create the most intelligent smart meters of all!
Extreme Market Pressure
The customers for the subject of this case study were power utilities. In many cases, governments in many parts of the world provided these utilities with incentives to switch to smart meters. At the same time, these smart meters had a long life span of 10 to 20 years, thus, once a utility selected a vendor, that customer would be out of the market for quite a long time. There were several start-up smart metering vendors and only a fixed number of big utilities. This created a land grab situation where each vendor was rushing to capture as many big utilities as they could. Whoever managed to garner the largest chunk of early market shares would dominate the market for a LONG time. This combination of incentives, long-term contracts, and limited customer base resulted in extremely high market pressure.
Combined, these extremes of market pressure and innovation created a hyper-accelerated market. One in which innovation and market pressure are so great that they result in a virtual hurricane for vendors to navigate.
What Is the Solution?
Transitioning to, adopting, and implementing agile methodologies in a software organization is a costly and time-consuming proposition. Companies and organizations need to know where they stand in terms of adoption/assimilation and its impact to the external agility of the business. They also need a framework to manage their performance so that strategic adjustments can be made. We set out to conduct studies that would provide greater understanding with regard to how agility can be managed and the impact of assimilation and adoption on these management processes. As we will see in a review of the latest research literature, most embedded systems environments, such as the one mentioned previously, with larger, more mature organizations, have taken a hybridized approach to agile adoption by combining agile with existing stage-gate methodologies. The orchestration of the processes required to support agile principles in such an environment and the way those principles are affected can reveal new insights into agility in new and different contexts. Such contexts include:
Geographically distributed teams,
Hybrid agility evolution, and
Embedded systems (environments which include the synchronization of firmware, software, and hardware development).(Continues…)
Excerpted from "Metagility"
Copyright © 2019 David A. Bishop.
Excerpted by permission of J. Ross Publishing, Inc..
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
Table of Contents
About the Author xv
Part 1 The Need for Metagility 1
Chapter 1 Agile Methods Today: What Has Changed and Why 3
The Agile Manifesto Today 4
Individuals and Interactions over Processes and Tools 5
Working Software over Comprehensive Documentation 6
Customer Collaboration over Contract Negotiation 6
Responding to Change over Following a Plan 7
Agile Management Today: A Research Perspective 8
Why Are Agile Methods and Agility Concepts so Critical to Business Success Now? 9
Hyper-Accelerated Markets 9
Approach for Building a New Direction 11
The Evolution of Agility: Engineering and Software Traditions 13
Agile Software Development Methodologies 17
How Have Most Companies Dealt with These Issues? 21
Achieving Agile Vorticity 22
Chapter 2 The Problem: Performance and Limitations of Agile Methods 23
Understanding the Performance of Agile Methods in Challenging Situations 23
Agile's Positive Contributions 24
Agile's Challenges 25
Managing Agility with Embedded and Real-Time Systems 27
Why Are Embedded Systems so Important? 29
Continuous Development and Managing Complex Change 29
Managing Distributed Teams 31
Adoption of Agility: The Need for Agile Transformation 32
Agile Has Become the Industry Standard for Managing ISD Efforts 32
Agile Adoption in Large, Complex Environments: Leveraging a Hybrid Approach 34
Problems with the Current Literature: Limitations of Agile Research 36
What Is Agile, Really? 36
An Apparent Lack of Rigor 37
No Research in Differing Contexts 37
Lack of Research at the Organizational Level or Across Business Units 37
Chapter 3 Hybrid Agility: Determining and Implementing the Right Approach 41
Why Organizations Adopt Some Aspects of Agility and Not Others, and How This Is Changing Agile Methods 41
Agile Is Not Just a Development Process Anymore, Its a New Way of Doing Business 44
The Agile Triangle Replacing the Iron Triangle 44
Obtaining Executive Involvement and Buy-In 45
Old Habits Die Hard: Convincing Others to Break Away from Waterfall 48
Agile Requires Too Many Meetings 48
We Don't Want to Provide High-Level Estimates 49
Agile Does Not Provide for Sound Technical or Architectural Planning 49
Agile Requires Colocation 50
Agile Does Not Allow for Long-Term Planning. 50
Our Product and/or Development Environment Is Too Complicated for Agile 51
Unspoken Concerns 51
Analyzing Your Environment to Determine the Correct Hybrid Agile Approach 51
How Waterfall Is Commonly Blended with Agile Methods 55
Stage-Gate Governed Release Train 58
Hybrid Agile Stage-Gate Definitions 58
Hybrid Agile for Firmware Teams 61
Hybrid Agile for Hardware Teams 62
Part 2 The Agile Business Vortex 63
Chapter 4 Understanding Agile Vortices 65
The Science Behind the Theory 66
Basic Design of the Research 66
How Data Was Collected 67
Qualitative Grounded Theory Analysis 68
Agile Vortices: How and Why They Occur 69
A Vignette of a Case Study 70
Converging Forces 74
The Agile Vortex Thought Experiment 75
Understanding the Systems Release 80
The Creation of Business Momentum 81
Agile Vorticity: The Key Connection Point 81
The Value of Agile Vortex Theory to Complex Contexts 83
Chapter 5 Managing Market Pressure 85
Market Agility 85
Establishing an Agile Market Share 86
How Does Your Customer Base Impact Business Agility? 88
Dealing with Government Regulation 91
Governing Market Pressures by Establishing a Strategic Direction 93
Customer Appetite: Maximizing Collaboration 95
Chapter 6 Product Genesis: Determining What to Build 97
Market Timing: How to Know if This Is the Right Time for Your Product 97
Dynamic Priorities: Managing Ever-Changing Priorities 99
Common Challenges with Dynamic Priorities 101
Requirements Comprehension: Understanding What the Market Needs 103
Decomposition: How to Break Down Requirements into Consumable Chunks 104
Functionality Chunking 106
Determining Which Requirements to Keep and Which Ones to Drop 110
Customer Needs 111
Customer-Driven Priorities 112
How to Negotiate Scope with Your Customers 114
Field Trials 115
Pilot Projects 116
Release Acceptance 117
Cultural Differences Affect Negotiation 117
Creation of the Roadmap 119
Instilling a Culture of Agility 120
A Recap of Business Momentum 121
Chapter 7 Organizational (Process) Agility: Building the Capability 123
How Do Hybrid Agile Implementations Work? Are They Successful? 123
How Is Agile Adopted Differently Between Software, Firmware, and Hardware Development? 125
What Is Different About Agile in Software Development and How Can This Difference Be Leveraged? 127
Managing Shared Resources: Resolving Conflicts and Getting the Job Done 127
Case Study Interview: Firmware in Embedded Systems Development 128
Is Waterfall Best in Certain Situations? 151
Prototyping to Accelerate Product Development 153
Making Use of Fast Tracking 154
Managing Customer Expectations for Agile Vorticity 155
Customer Negotiation Techniques for Maintaining Business Momentum 155
What Situations Benefit Most from a Hybrid Agile Mix? 157
Part 3 Practical Issues and Applications 159
Chapter 8 The Art of Metagility: Bringing It All Together 161
Metagility in Complex Environments 161
Embedded Systems: A Case Study in Agile Orchestration 161
Interconnections and Interactions 162
Managing One-Way Dependencies 163
Intertwined Solutions: Managing Interdependencies 164
Defining Formal Linkages: Developing Productive and Impactful Meetings 165
Metagility Metrics: Predictive Analytics Dashboards for Executives (Status Points) 172
Decision Points: The What, When, and Where of Metagility Decision Making 181
Informal Interactions for Maximum Agility: Eliminating Forgetfulness 182
Fine Tuning: How to Optimize 183
Customer Acceptance 184
Scope Adjustment 185
Resource Adjustment 186
Kaizen Culture: The Importance of Constant Reassessment 187
Chapter 9 Soft Skills for Managing Agility 189
Addressing Common Project Management Concerns 189
Team Management 191
Understanding Tribes and Building Agile Teams 192
Self-Organizing Teams 194
Managing Distributed Teams 195
Interchangeable Teams 197
Techniques for Bringing Your Team up to Speed 198
Teams Are Not Equal 198
Pair Programming 199
Executive Management: Managing Your Managers 199
Real-Time Strategic Alignment 200
The Number One Secret Ingredient to Becoming and Remaining Agile 211
Chapter 10 The Future of Agile Methods 215
An Inquiry into Hybrid Agility 215
Fluidity and Continuous Releases: The Cause for Leaner Methodologies 216
Hybrid Agile Implementations: Whirlpools within a River of Innovation 217
Implications for Research 219
Implications for Practice 220
Trends: Looking Ahead 221
Data Science Along with Advanced Tooling Will Drive Decisions 221
Approaches to Communication Methods Must Be Continually Re-evaluated 222
Agile Organizations Require Agile Individuals 222
Agile Companies Require Agile Leaders 223
Agile Is Becoming Integral to Strategic Planning 223
A Caution Regarding Agile Frameworks 224
Agile Evolution and Natural Selection 225
New Markets Must Be Continually Sought Out 225
Agility Will Continue to Evolve 226