Read an Excerpt

Software Cost Estimation and Sizing Methods

Rand Corporation

Chapter One

The Air Force Cost Analysis Agency (AFCAA) supports the Air Force Secretariat by conducting independent component cost analyses, special cost reviews, and cost-analysis research. To these ends, AFCAA is organized as four separate estimating divisions and one cost-research division:

Aircraft and Weapons Division, with the mission of developing estimates to support proposed aircraft, guided weapons, and missile systems. Information Technology Division, with the mission of developing estimates for information technology projects. Space Technology Division, with the mission of developing estimates for space-based Air Force projects. Force Analysis Division, with the mission of developing factors and performing estimates focused on long-range planning. This division also develops and maintains the Air Force Total Ownership Cost (AFTOC) database, a repository of historical information about estimates and costs that is useful in informing future cost estimates and cost-related decisions. Research and Resource Management Division, which provides support to these technical divisions.

The analysts in the technical divisions use a host of estimation tables and tools to generate cost and schedule estimates for hardware and software. Alldivisions have a common interest in improving software-estimating tools and in producing more-accurate software cost estimates.

Every cost estimate is inherently risky, in that an analyst must predict likely cost when there is much unknown about the software to be built. That is, since a risk is a problem that may occur during development, the analyst is asked to anticipate problems with development before development has begun. At the stage at which the estimate is being made, it may not even be known which staff will build and test the software, or what kind of design will solve the problem described by the software requirements. Thus, the notion of risk is central to any cost analysis. In particular, since the size of a software system is uncertain until development is completed, the risk of estimating size incorrectly is a major component of the risk of inaccurate cost estimates.

This document focuses on the role of risk in producing size estimates (used as inputs to software cost and schedule estimates) and the cost estimates themselves. Intended for use by experienced cost analysts, the document addresses how to manage the risks inherent in selecting and using a particular estimation method, especially at the early stages of a project, when many of the factors needed to support estimation may be unknown or uncertain.

Rather than seeking a universal, one-size-fits-all size- or cost-estimation method, this report supports a careful analysis of factors that affect the accuracy of the estimate. In particular, the two key factors addressed in this report are the decisions made during the estimation process (such as which methods and models are most appropriate for a given situation) and the nature of the data (such as software size) used in the estimation process.

Two techniques can improve accountability of risks relating to software estimates: identifying areas of uncertainty (that may lead to risky situations) and analyzing the estimation process to determine where risk mitigation can reduce the uncertainty as the estimate is produced. The first technique increases an analyst's diligence in reporting uncertainty by recognizing that uncertainty is inherent in any estimation but oftentimes goes unreported. For example, managers sometimes expect to be given a point estimate-the software will cost x dollars-when, in fact, the best that can be said is that the cost will lie within a given feasible range, characterized by a particular distribution. The distribution itself expresses the degree of uncertainty in the estimate: A narrow distribution is less uncertain than a wide one, and the distribution's shape imparts additional information about the likely cost. The capture, quantification, and reporting of that uncertainty give greater credibility to the estimation.

The second technique is to actually address and mitigate risks in the estimation process, thereby reducing the total uncertainty and increasing the estimate's precision. This technique examines the estimation process itself to identify each choice at each decision point in what makes up a decision tree. Then, each point is analyzed to determine what risks are inherent in making each choice. Every possible path through the decision tree represents a set of risks, and management can associate with each path a set of actions to mitigate the risks whenever possible.

The two techniques are complementary. The first technique improves accountability by reporting the uncertainty. The second technique improves accountability by dealing with and reducing the uncertainty. This document addresses both techniques, offering guidelines to cost analysts on how best to manage the unavoidable risks that are attendant on predicting software size and cost. These techniques inject realism into the estimation process, acknowledging that estimates are often made with limited knowledge of the system and from a profusion of choices that may be rife with uncertainty.

Study Methodology

This study's primary objectives were twofold: to provide a general assessment of the risks involved in software cost estimation and to provide strategies to mitigate those risks. In support of these objectives, the research contained four tasks, each of which was accomplished by literature review and analysis. Where applicable, the reviews incorporated lessons from sample programs to develop pragmatic risk-mitigation strategies. The four tasks were (1) understanding the concept of risk as it applies to software cost estimation, (2) examining why and how risk occurs in software estimates, (3) detailing the options for choosing estimation techniques and their required inputs when developing estimates, and (4) developing strategies and options to mitigate risks. We describe each task in turn.

1. Risk and Software Cost Estimation

We began by establishing a taxonomy for terms such as uncertainty, error, risk, and accuracy. Risk is usually expressed in the form of confidence levels and confidence limits. Confidence level refers to a statistical percentage of certainty; for example, "At the 95-percent confidence level, we can say that our cost falls between X and Y." This statement expresses the fact that, statistically, there is a 95-percent chance that the true cost of the system falls between the given confidence limits X and Y. This expression of risk is an indication of how accurate the estimate is believed to be. However, the factors that drive this accuracy (or lack thereof) are uncertainty and error. Thus, understanding the sources of error and uncertainty was addressed in the second task.

2. Sources of Risk in Software Estimates

We explored the relationships among risk, error, and uncertainty by asking the question, "Where are the sources of error in software cost and schedule estimates?" By decomposing software estimation into three basic components-select method, collect data, and apply method-we developed a basic model for error. Error can be introduced with the data (that is, with the estimation model's inputs) or with the estimation process (that is, by way of the decisions that are influenced by the system definition, system development, and the estimation method). We used existing literature and program experiences to characterize these errors and the underlying uncertainties. Two areas-sizing software and using cost models-warranted detailed descriptions to facilitate the application of risk-mitigation strategies. They were addressed in Task 3.

3. Options in Developing Estimates

We described each of the major approaches to sizing and cost estimation, so that the analyst could compare and contrast the methods when making a decision about which method is most appropriate (or comparing the results from two different methods). Our descriptions are based largely on our review of existing literature.

4. Strategies for Risk Mitigation

We concluded this study by developing checklists to codify the recommended practices, guidance, and lessons learned for reducing the uncertainty and errors in the areas of risk that we identified. Because the checklists address the similarities among software projects but not their differences, they are neither complete nor comprehensive. Rather, they provide a useful framework and an initial set of strategies upon which to build. They must be augmented periodically by new research and actual program experience. Other augmentations might involve additional quantitative information to help an analyst determine which risks are more likely or are of greater consequence.

Report Organization

To describe and analyze the risks, this document is organized in several chapters. Chapter Two begins with a discussion of two important issues related to estimation: the meaning of estimation quality, and the differences among the concepts of error, risk, and uncertainty. Following that discussion is a description of some of the major issues analysts must consider when selecting and using a sizing method. Then Chapter Three focuses on current sizing methods, including what each is, how to use it, and what its output is likely to be. The document contrasts the different methods, laying out their pros and cons in terms of such issues as how much uncertainty is inherent in using the method, and whether the method relies on historical data or previous experience. In Chapter Four, the issues, the pros and cons of each method, and the information about risks and uncertainties are then organized as a risk checklist, so that an analyst can see what risks are inherent in choosing a particular sizing method and in using it in a particular way. This checklist can be applied to an existing or proposed sizing method to help assess its appropriateness or usefulness in a given situation. Chapter Five presents the risks again, reorganized to help an analyst review an existing size estimate and determine whether the estimate has addressed all relevant issues.

The remainder of the document addresses the more global risks inherent in cost estimation. Cost analysts must produce software cost estimates for a variety of programs. Ideally, the estimated cost will prove to be very close to the actual cost, and several estimation models purport to be flexible enough to provide accurate estimates in almost any situation. Unfortunately, this ideal is far from the reality. In fact, some models work best when restricted to particular situations or when applied using databases tailored to an organization's particular experiences.

A more realistic approach to software cost estimation involves understanding the differences among models as well as the risks inherent in selecting one model over another. The findings highlighted in this report are intended to aid cost analysts in managing the risks inherent in providing software cost estimates, in two key ways:

1. By discussing the pros and cons of various cost-estimation methods

2. By providing a concise risk checklist that can be applied to an existing or proposed cost-estimation method to help assess its appropriateness in a given situation.

To these ends, Chapter Six describes the characteristics of different techniques used to provide estimates and the role of historical databases in supporting estimation. Focusing on the risks inherent in cost estimation, Chapter Seven details the sources of risks and errors, presenting risk-related checklists that cost analysts can use in performing an estimate and for evaluating the estimates of others. Finally, Chapter Eight summarizes the way that cost analysts can use the notion of risk to guide estimation decisions.

Chapter Two

Given that size is the principal determinant of cost in most cost-estimation models, an accurate size estimate is crucial to good cost estimation. We explore the several major approaches to sizing and the different sizing methods in Chapter Three, noting here that many of the methods share common advantages and disadvantages. In this chapter, we discuss several aspects of size estimation and the pros and cons to be considered when selecting a method. This discussion makes visible the issues involved when identifying the risks inherent in estimating size. Chapters Four and Five build on this basis, by highlighting risks and suggesting strategies to mitigate them.

Because accurate size estimates mitigate risk more than any other cost-related parameter, it is important that software sizing be done as consistently and accurately as possible, given the uncertainties inherent in estimation. Although analysts like to think that they can remove risk by improving accuracy, the more realistic hope is that analysts learn to manage risk by understanding, exploring, and (wherever possible) reducing the uncertainties involved in producing estimates.

Several issues make software sizing difficult. First, software sizing is performed in a variety of different contexts, some with a great deal of knowledge about the system and some with almost no knowledge at all. For a new project, the sizing estimate must be done at or near the very beginning of a project, before the actual software is written and, often, before the requirements are finalized. In this case, sizing usually involves some kind of translation from entities, characteristics, and relationships in the requirements or design to the likely size of the code once it is written. The nature and correctness of the translation are major factors that need to be addressed.

For projects that have already started, some of the software may already be written but require additions, changes, and deletions, as in a major satellite development program in which the system was already four years into development when AFCAA was asked to estimate the cost of completing the software. In such cases, and when operational software is being maintained, the sizing estimate must take into account not only how much software must be changed but also how knowledgeable the developers are. When the maintainer is not the person who built the original software, changes may take longer; the maintainer needs time to understand the existing requirements, design, and code before designing and implementing changes. Thus, the sizing estimate must include not only the direct changes but also any "scaffolding" software needed to evaluate, change, and test the existing software.

In the middle of a project, it is useful to examine the immediate prior history of that project to manage the expectations of the remainder. For example, when analysts on the above satellite system considered the projections for the size of the software at completion, the contractor-supplied estimates included a significant amount of reuse. In particular, at project start, a tremendous amount of reuse was predicted, based on the claim that "it was the same system; it was just being rehosted." However, the originally predicted levels of reuse were not achievable; the software size grew considerably as the system was implemented. This past experience was useful in determining future projections about the size of the remaining software; as a consequence, the predicted levels of remaining reuse were reduced.

(Continues...)

---

Excerpted from Software Cost Estimation and Sizing Methods by Shari Lawrence Pfleeger Felicia Wu Rosalind Lewis Copyright © 2005 by RAND Corporation. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

---